Month: October 2025

In Part 1 of our series, we introduced a proactive cost management solution for Amazon Bedrock, featuring a robust cost sentry mechanism designed to enforce real-time token usage limits. We explored the core architecture, token tracking strategies, and initial budget enforcement techniques that help organizations control their generative AI expenses.
Building upon that foundation, this post explores advanced cost monitoring strategies for generative AI deployments. We introduce granular custom tagging approaches for precise cost allocation, and develop comprehensive reporting mechanisms.
Solution overview
The cost sentry solution introduced in Part 1 was developed as a centralized mechanism to proactively limit generative AI usage to adhere to prescribed budgets. The following diagram illustrates the core components of the solution, adding in cost monitoring through AWS Billing and Cost Management.

Invocation-level tagging for enhanced traceability
Invocation-level tagging extends our solution’s capabilities by attaching rich metadata to every API request, creating a comprehensive audit trail within Amazon CloudWatch logs. This becomes particularly valuable when investigating budget-related decisions, analyzing rate-limiting impacts, or understanding usage patterns across different applications and teams. To support this, the main AWS Step Functions workflow was updated, as illustrated in the following figure.

Enhanced API input
We also evolved the API input to support custom tagging. The new input structure introduces optional parameters for model-specific configurations and custom tagging:

{
  “model”: “string”,     // e.g., “claude-3” or “anthropic.claude-3-sonnet-20240229-v1:0”
  “prompt”: {
    “messages”: [
      {
        “role”: “string”,    // “system”, “user”, or “assistant”
        “content”: “string”
      }
    ],
    “parameters”: {
      “max_tokens”: number,    // Optional, model-specific defaults
      “temperature”: number,   // Optional, model-specific defaults
      “top_p”: number,         // Optional, model-specific defaults
      “top_k”: number          // Optional, model-specific defaults
    }
  },
  “tags”: {
    “applicationId”: “string”,  // Required
    “costCenter”: “string”,     // Optional
    “environment”: “string”     // Optional – dev/staging/prod
  }
}

The input structure comprises three key components:

model – Maps simple names (for example, claude-3) to full Amazon Bedrock model IDs (for example, anthropic.claude-3-sonnet-20240229-v1:0)
input – Provides a messages array for prompts, supporting both single-turn and multi-turn conversations
tags – Supports application-level tracking, with applicationId as the required field and costCenter and environment as optional fields

In this example, we use different cost centers for sales, services, and support to simulate the use of a business attribute to track usage and spend for inference in Amazon Bedrock. For example:

{
  “model”: “claude-3-5-haiku”,
  “prompt”: {
    “messages”: [
      {
        “role”: “user”,
        “content”: “Explain the benefits of using S3 using only 100 words.”
      },
      {
        “role”: “assistant”,
        “content”: “You are a helpful AWS expert.”
      }
    ],
    “parameters”: {
      “max_tokens”: 2000,
      “temperature”: 0.7,
      “top_p”: 0.9,
      “top_k”: 50
    }
  },
  “tags”: {
    “applicationId”: “aws-documentation-helper”,
    “costCenter”: “support”,
    “environment”: “production”
  }
}

Validation and tagging
A new validation step was added to the workflow for tagging. This step uses an AWS Lambda function to add validation checks and maps the model requested to the specific model ID in Amazon Bedrock. It supplements the tags object with tags that will be required for downstream analysis.
The following code is an example of a simple map to get the appropriate model ID from the model specified:

MODEL_ID_MAPPING = {
    “nova-lite”: “amazon.nova-lite-v1:0”,
    “nova-micro”: “amazon.nova-micro-v1:0”,
    “claude-2”: “anthropic.claude-v2:0”,
    “claude-3-haiku”: “anthropic.claude-3-haiku-20240307-v1:0”,
    “claude-3-5-sonnet-v2”: “us.anthropic.claude-3-5-sonnet-20241022-v2:0”,
    “claude-3-5-haiku”: “us.anthropic.claude-3-5-haiku-20241022-v1:0”
}

Logging and analysis
By using CloudWatch metrics with custom-generated tags and dimensions, you can track detailed metrics across multiple dimensions such as model type, cost center, application, and environment. Custom tags and dimensions show how teams use AI services. To see this analysis, steps were implemented to generate custom tags, store metric data, and analyze metric data:

We include a unique set of tags that capture contextual information. This can include user-supplied tags as well as ones that are dynamically generated, such as requestId and timestamp:

  “tags”: {
    “requestId”: “ded98994-eb76-48d9-9dbc-f269541b5e49”,
    “timestamp”: “2025-01-31T14:05:26.854682”,
    “applicationId”: “aws-documentation-helper”,
    “costCenter”: “support”,
    “environment”: “production”
}

As each workflow is executed, the limit for each model will be evaluated to make sure the request is within budgetary guidelines. The workflow will end based on three possible outcomes:

Rate limit approved and invocation successful
Rate limit approved and invocation unsuccessful
Rate limit denied
The custom metric data is saved in CloudWatch in the GenAIRateLimiting namespace. This namespace includes the following key metrics:

TotalRequests – Counts every invocation attempt regardless of outcome
RateLimitApproved – Tracks requests that passed rate limiting checks
RateLimitDenied – Tracks requests blocked by rate limiting
InvocationFailed – Counts requests that failed during model invocation
InputTokens – Measures input token consumption for successful requests
OutputTokens – Measures output token consumption for successful requests
Each metric includes dimensions for Model, ModelId, CostCenter, Application, and Environment for data analysis.
We use CloudWatch metrics query capabilities with math expressions to analyze the data collected by the workflow. The data can be displayed in a variety of visual formats to get a granular view of requests by the dimensions provided, such as model or cost center. The following screenshot shows an example dashboard that displays invocation metrics where one model has reached its limit.

Additional Amazon Bedrock analytics
In addition to the custom metrics dashboard, CloudWatch provides automatic dashboards for monitoring Amazon Bedrock performance and usage. The Bedrock dashboard offers visibility into key performance metrics and operational insights, as shown in the following screenshot.

Cost tagging and reporting
Amazon Bedrock has introduced application inference profiles, a new capability that organizations can use to apply custom cost allocation tags to track and manage their on-demand foundation model (FM) usage. This feature addresses a previous limitation where tagging wasn’t possible for on-demand FMs, making it difficult to track costs across different business units and applications. You can now create custom inference profiles for base FMs and apply cost allocation tags like department, team, and application identifiers. These tags integrate with AWS cost management tools including AWS Cost Explorer, AWS Budgets, and AWS Cost Anomaly Detection, enabling detailed cost analysis and budget control.
Application inference profiles
To start, you must create application inference profiles for each type of usage you want to track. In this case, the solution defines custom tags for costCenter, environment, and applicationId. An inference profile will also be based on an existing Amazon Bedrock model profile, so you must combine the desired tags and model into the profile. At the time of writing, you must use the AWS Command Line Interface (AWS CLI) or AWS API to create one. See the following example code:

aws bedrock create-inference-profile
  –inference-profile-name “aws-docs-sales-prod”
  –model-source ‘{“copyFrom”:  “arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-haiku-20240307-v1:0”}’
  –tags ‘[
    {“key”: “applicationId”, “value”: “aws-documentation-helper”},
    {“key”: “costCenter”, “value”: “sales”},
    {“key”: “environment”, “value”: “production”}
  ]’

This command creates a profile for the sales cost center and production environment using Anthropic’s Claude Haiku 3.5 model. The output from this command is an Amazon Resource Name (ARN) that you will use as the model ID. In this solution, the ValidateAndSetContext Lambda function was modified to allow for specifying the model by cost center (for example, sales). To see which profiles you created, use the following command:
aws bedrock list-inference-profiles –type-equals APPLICATION
After the profiles have been created and the validation has been updated to map cost centers to the profile ARNs, the workflow will start running inference requests with the aligned profile. For example, when the user submits a request, they will specify the model as sales, services, or support to align with the three cost centers defined. The following code is a similar map to the previous example:

MODEL_ID_MAPPING = {
    “sales”: “arn:aws:bedrock:<region>:<account>:application-inference-profile/<unique id1>”,
    “services”: “arn:aws:bedrock:<region>:<account>:application-inference-profile/<unique id2>”,
    “support”: “arn:aws:bedrock:<region>:<account>:application-inference-profile/<unique id3>”
   }

To query CloudWatch metrics for the model usage correctly when using application inference profiles, you must specify the unique ID for the profile (the last part of the ARN). CloudWatch will store metrics like token usage based on the unique ID. To support both profile and direct model usage, the Lambda function was modified to add a new tag for modelMetric to be the appropriate term to use to query for token usage. See the following code:

  “tags”: {
    “requestId”: “ded98994-eb76-48d9-9dbc-f269541b5e49”,
    “timestamp”: “2025-01-31T14:05:26.854682”,
    “applicationId”: “aws-documentation-helper”,
    “costCenter”: “support”,
    “environment”: “production”,    
    “modelMetric”: “<unique id> | <model id>”
  }

Cost Explorer
Cost Explorer is a powerful cost management tool that provides comprehensive visualization and analysis of your cloud spending across AWS services, including Amazon Bedrock. It offers intuitive dashboards to track historical costs, forecast future expenses, and gain insights into your cloud consumption. With Cost Explorer, you can break down expenses by service, tags, and custom dimensions, for detailed financial analysis. The tool updates on a daily basis.
When you use application inference profiles with Amazon Bedrock, your AI service usage is automatically tagged and flows directly into Billing and Cost Management. These tags enable detailed cost tracking across different dimensions like cost center, application, and environment. This means you can generate reports that break down Amazon Bedrock AI expenses by specific business units, projects, or organizational hierarchies, providing clear visibility into your generative AI spending.
Cost allocation tags
Cost allocation tags are key-value pairs that help you categorize and track AWS resource costs across your organization. In the context of Amazon Bedrock, these tags can include attributes like application name, cost center, environment, or project ID. To activate a cost allocation tag, you must first enable it on the Billing and Cost Management console. After they’re activated, these tags will appear in your AWS Cost and Usage Report (CUR), helping you break down Amazon Bedrock expenses with granular detail.
To activate a cost allocation tag, complete the following steps:

On the Billing and Cost Management console, in the navigation pane, choose Cost Allocation Tags.
Locate your tag (for this example, it’s named costCenter) and choose Activate.
Confirm the activation.

After activation, the costCenter tag will appear in your CUR and will be used in Cost Explorer. It might take 24 hours for the tag to become fully active in your billing reports.

Cost Explorer reporting
To create an Amazon Bedrock usage report in Cost Explorer based on your tag, complete the following steps:

On the Billing and Cost Management console, choose Cost Explorer in the navigation pane.
Set your desired date range (relative time range or custom period).
Select Daily or Monthly granularity.
On the Group by dropdown menu, choose Tag.
Choose costCenter as the tag key.
Review the displayed Amazon Bedrock costs broken down by each unique cost center value.
Optionally, filter the values by applying a filter in the Filters section:

Choose Tag filter.
Choose the costCenter tag.
Choose specific cost center values you want to analyze.

The resulting report will provide a detailed view of Amazon Bedrock AI service expenses, helping you compare spending across different organizational units or projects with precision.

Summary
The AWS Cost and Usage Reports (including budgets) act as trailing edge indicators because they show what you’ve already spent on Amazon Bedrock after the fact. By blending real-time alerts from Step Functions with comprehensive cost reports, you can get a 360-degree view of your Amazon Bedrock usage. This reporting can alert you before you overspend and help you understand your actual consumption. This approach gives you the power to manage AI resources proactively, keeping your innovation budget on track and your projects running smoothly.
Try out this cost management approach for your own use case, and share your feedback in the comments.

About the Author
Jason Salcido is a Startups Senior Solutions Architect with nearly 30 years of experience pioneering innovative solutions for organizations from startups to enterprises. His expertise spans cloud architecture, serverless computing, machine learning, generative AI, and distributed systems. Jason combines deep technical knowledge with a forward-thinking approach to design scalable solutions that drive value, while translating complex concepts into actionable strategies.

As organizations embrace generative AI powered by Amazon Bedrock, they face the challenge of managing costs associated with the token-based pricing model. Amazon Bedrock offers a pay-as-you-go pricing structure that can potentially lead to unexpected and excessive bills if usage is not carefully monitored. Traditional methods of cost monitoring, such as budget alerts and cost anomaly detection, can help spot unexpectedly high usage but are reactive in nature. To address costs proactively, it is vital to use both leading and trailing indicators.
Leading indicators are predictive signals that help you anticipate future trends or potential issues before they fully materialize. These indicators provide proactive insights that allow for timely intervention. In contrast, trailing indicators are retrospective measurements that confirm what has already occurred. By understanding and tracking both types of indicators, organizations can develop more strategic and responsive decision-making processes.
In this two-part series, we introduce a comprehensive solution for proactively managing Amazon Bedrock inference costs. Our approach features a cost sentry mechanism designed to establish and enforce token usage limits, providing organizations with a robust framework for controlling generative AI expenses. In this post, we focus on core architecture, cost sentry design, token usage tracking, and initial budget enforcement strategies. In Part 2, we explore advanced monitoring techniques, custom tagging, reporting, and long-term cost optimization best practices. The goal is to deliver a predictable, cost-effective approach to Amazon Bedrock deployments that aligns with organizational financial constraints.
Solution overview
Amazon Bedrock is billed on a token usage-based policy with charges based on the input and output tokens used. The rate charged depends on the model used and AWS Region where inference is performed. Developers must implement robust token management strategies in their applications to help prevent runaway costs, making sure generative AI applications include circuit breakers and consumption limits that align with budgetary constraints.
To address this, you can configure Amazon CloudWatch alarms or monitor costs with billing alerts and budgets, but these mechanisms look at incurred costs or usage after the fact. Another option is the Generative AI Gateway Solution in the AWS Solutions Library, which uses LiteLLM to enforce budgetary limits for Amazon Bedrock and other model providers.
This solution was developed to identify a proactive, centralized mechanism that could limit the generative AI usage to a specific budget that can be adjusted. This approach uses serverless workflows and native Amazon Bedrock integration that offers less operational complexity while providing large-scale performance and scaling.
When building applications with Amazon Bedrock, it is common practice to access the service through a developed API, either synchronously through a REST API or asynchronously through a queuing system. The following diagram compares these architectures.

For synchronous interactions, clients make direct REST API calls to Amazon Bedrock, passing in the necessary parameters. In an asynchronous architecture, clients submit inference requests to a queue or message broker, such as Amazon Simple Queue Service (Amazon SQS). A backend processing system, often implemented as a serverless function or a containerized application, continuously monitors the queue and processes incoming requests. This approach decouples the client from the inference processing, enabling scalability and resilience in handling bursts of requests.
This solution is a centralized mechanism that can be used to interact with Amazon Bedrock to serve as a proactive cost sentry. It is designed using a serverless architecture that uses AWS Step Functions to orchestrate a workflow that validates token usage against configured limits before allowing Amazon Bedrock inference requests to proceed. This solution makes sure that generative AI applications stay within predefined budgetary boundaries, providing cost predictability and control.
The following diagram illustrates the architecture we build in this post.

The core components of this solution include:

Rate limiter workflow – A Step Functions workflow that retrieves current token usage metrics from CloudWatch, compares them against predefined limits stored in Amazon DynamoDB, and determines whether to proceed with or deny the Amazon Bedrock inference request.
Amazon Bedrock model router – A separate Step Functions state machine that acts as a centralized gateway for invoking various Amazon Bedrock models. This component abstracts the complexity of handling different I/O parameters required by each model.
Token usage tracking – Uses CloudWatch metrics integration with Amazon Bedrock to retrieve current token usage data for input and output tokens across all or specific models.
Budget configuration – Allows setting token usage limits on a per-model basis by storing the desired budget values in DynamoDB. A default limit can also be set to apply to models without specific budgets defined.
Cost and usage visibility – Provides visibility for AI usage with CloudWatch dashboards and cost over time reporting in AWS Cost Explorer.

The solution follows a serverless architecture approach, using managed AWS services like Step Functions, AWS Lambda, DynamoDB, and CloudWatch to provide a scalable, extensible, and cost-effective implementation.
The goal is to provide a proactive method of setting generative AI usage limits that operate as a leading indicator to limit usage:

Proactive budgeting – Enforces token usage limits before allowing inference requests, helping prevent accidental overspending
Model-specific budgets – Supports setting individual budgets for different Amazon Bedrock models based on their pricing and usage patterns
Default budget fallback – If no specific budget is defined for a model, a default limit can be applied to enable cost control
Monitoring – Uses CloudWatch metrics integration to track token usage, enabling accurate budget enforcement
Serverless architecture – Uses Step Functions, Lambda, DynamoDB, and CloudWatch for a scalable and cost-effective solution
Extensibility – The modular design allows for seamless integration of additional Amazon Bedrock models or alternative inference methods

Step Functions workflows
In this section, we explore how the solution uses Step Functions to implement rate limiting and model routing workflows.
Rate limiting workflow
The rate limiting workflow is designed to take a minimal JSON document as input with the following format:

{
  “modelId”: “string”,       // e.g. “anthropic.claude-3-sonnet-20240229-v1:0”
  “prompt”: {
    “messages”: [
      {
        “role”: “string”,    // “system”, “user”, or “assistant”
        “content”: “string”
      }
    ]
  }
}

This workflow is the core component that enforces budgetary controls. The key steps are as follows:

A Lambda function retrieves the start and end dates for the current month, which is used to query token usage metrics for the appropriate time range.
The workflow queries CloudWatch to retrieve the current month’s token usage metrics for the specified Amazon Bedrock model.
The workflow retrieves the configured token usage limit for the specified Amazon Bedrock model from DynamoDB. If no specific limit is found, it falls back to retrieving the default limit.
The workflow compares the current token usage against the configured limit to determine if the budget has been exceeded or not.
If the token usage is within the budget, this step invokes the Amazon Bedrock model router state machine to perform the actual inference request.
Depending on the outcome of the budget check, the workflow returns either the formatted inference result or an error indicating that the budget has been exceeded.

The following diagram illustrates the Step Functions workflow.

Amazon Bedrock model router workflow
The Amazon Bedrock model router workflow is a separate Step Functions state machine responsible for invoking the appropriate Amazon Bedrock model based on the request parameters. It abstracts the complexity of handling different I/O formats required by various Amazon Bedrock models and combines the result into a standardized format.
The key steps in the workflow include:

Based on the provided model ID, the workflow determines the specific Amazon Bedrock model to be invoked.
The workflow calls the appropriate Amazon Bedrock model with the required input parameters.
The workflow normalizes the output from the Amazon Bedrock model to a consistent format for further processing or returning to the client.
The workflow returns the transformed inference result to the calling workflow (budget sentry workflow).

The following diagram illustrates the Step Functions workflow.

You can implement additional steps to handle error conditions and format the output appropriately. In this example, the Anthropic flow includes error processing.
Token usage tracking with CloudWatch metrics
The Amazon Bedrock cost sentry uses the CloudWatch integration with Amazon Bedrock to retrieve current token usage metrics. These metrics are used to enforce budgetary limits proactively. For example, see the following query:

{
    “sparkline”: false,
    “metrics”: [
        [ { “expression”: “SEARCH(‘{AWS/Bedrock} MetricName=”InputTokenCount”‘, ‘Sum’, 60)”, “region”: “us-east-1” } ],
        [ { “expression”: “SEARCH(‘{AWS/Bedrock} MetricName=”OutputTokenCount”‘, ‘Sum’, 60)”, “region”: “us-east-1” } ]
    ],
    “legend”: {
        “position”: “right”
    },
    “title”: “InputTokenCount, OutputTokenCount”,
    “region”: “us-east-1”,
    “liveData”: true,
    “view”: “gauge”,
    “stacked”: false,
    “period”: 2592000,
    “table”: {
        “summaryColumns”: [
            “SUM”
        ]
    },
    “yAxis”: {
        “left”: {
            “min”: 0,
            “max”: 1000000
        }
    },
    “setPeriodToTimeRange”: true,
    “trend”: false,
    “startTime”: “2024-05-01T00:00:00Z”,
    “endTime”: “2024-05-30T23:59:59Z”
}

This CloudWatch metric query retrieves the total input and output token counts for a specified time range, allowing the rate limiter workflow to accurately enforce budgets based on real-time usage data.
Budget configuration with DynamoDB
The Amazon Bedrock cost sentry stores token usage limits in a DynamoDB table, providing seamless configuration and updates to individual model budgets or the default limit. For example, see the following code:

{
    “modelId”: “anthropic.claude-3-sonnet-20240229-v1:0”,
    “limit”: {
        “input”: 1000000,
        “output”: 3000000
    }
}

In this example, the token usage limit for the specified Amazon Bedrock model (anthropic.claude-3-sonnet-20240229-v1:0) is set to 1,000,000 input tokens and 3,000,000 output tokens.
Administrators can quickly update these limits by modifying the corresponding DynamoDB records, providing flexibility in adjusting budgets as needed.
Performance analysis of the rate limiter workflow
To assess the performance impact of introducing the workflow, we used an array of inference requests. Test cases included various prompts designed to generate responses ranging from concise answers to detailed explanations over 500 words, effectively testing the workflow’s performance across different output token sizes. The workflow demonstrated exceptional performance characteristics across 501 successful executions, handling a diverse set of inference requests from brief responses to extensive content generation.
The workflow maintains consistent execution patterns while processing requests ranging from 6.76 seconds to 32.24 seconds in total duration, with the variation primarily reflecting the different output token requirements of each request:

Quick responses (under 10 seconds) – Typically handling concise answers and simple queries
Medium-length content (11–22 seconds) – Common for detailed explanations and multi-paragraph responses
Extended generation (up to 32 seconds) – Handling comprehensive responses requiring more than 500 words

The following diagram illustrates our time distribution findings.

The time distribution analysis reveals highly optimized resource utilization:

Amazon Bedrock model router – 5.80–31.99 seconds (98.26% of runtime)
Other workflow steps – 0.11–4.74 seconds (1.65% of runtime)
System overhead – 0.02 seconds average (0.09% of runtime)

This performance profile aligns with best practices for workflow orchestration, where minimizing overhead and maintaining consistent execution patterns are crucial for reliability. The workflow’s efficiency is evidenced by its remarkably low system overhead of just 0.09%, demonstrating effective use of the built-in controls and state management capabilities of Step Functions regardless of the response size being generated.
The execution consistency is particularly noteworthy, with a predictable event pattern of 47–49 events per execution, regardless of the inference request complexity or output size. This predictability is essential for workload management and resource planning, especially when handling varied request complexities and token outputs.
These metrics indicate a well-architected workflow that effectively uses Step Functions Express workflow capabilities for high-volume event processing while maintaining minimal overhead and consistent performance characteristics across both simple queries and complex, token-intensive inference requests.
Cost analysis
To analyze the cost implications, estimates were generated using the AWS Pricing Calculator for both Standard and Express Step Functions workflows, assuming 100,000 requests per month. The following table summarizes these estimates.

Detailed Estimate

Region
Description
Service
Upfront
Monthly
First 12 Months Total
Currency
Configuration Summary

US East (Ohio)
Step Functions Standard
Step Functions – Standard Workflows
0
$37.40
$448.80
USD
Workflow requests (100,000 per month)State transitions per workflow (15)

US East (Ohio)
Step Functions Express
Step Functions – Express Workflows
0
$3.75
$45
USD
Duration of each workflow (35,000)Memory consumed by each workflow (64 MB)Workflow requests (100,000 per month)

The cost analysis revealed that the Step Functions Express workflow offers a more cost-effective solution compared to the Standard workflow, with potential cost savings of up to 90% for the same workload. There is a potential for cost reduction for Standard if the number of steps can be optimized. For example, a few formatting pass steps could potentially be removed, but these steps help format the downstream input to later steps.
Consult the AWS Pricing Calculator for more details on pricing and to run your own scenario.
Conclusion
In this solution, we used Step Functions to build a system that serves as a leading indicator because it tracks rate limiting and token usage, warning us immediately when we’re approaching our usage limits. In Part 2, we discuss combining this with trailing indicators to stay aware of usage and costs.

About the author
Jason Salcido is a Startups Senior Solutions Architect with nearly 30 years of experience pioneering innovative solutions for organizations from startups to enterprises. His expertise spans cloud architecture, serverless computing, machine learning, generative AI, and distributed systems. Jason combines deep technical knowledge with a forward-thinking approach to design scalable solutions that drive value, while translating complex concepts into actionable strategies.

Deploying machine learning (ML) models into production can often be a complex and resource-intensive task, especially for customers without deep ML and DevOps expertise. Amazon SageMaker Canvas simplifies model building by offering a no-code interface, so you can create highly accurate ML models using your existing data sources and without writing a single line of code. But building a model is only half the journey; deploying it efficiently and cost-effectively is just as crucial. Amazon SageMaker Serverless Inference is designed for workloads with variable traffic patterns and idle periods. It automatically provisions and scales infrastructure based on demand, alleviating the need to manage servers or pre-configure capacity.
In this post, we walk through how to take an ML model built in SageMaker Canvas and deploy it using SageMaker Serverless Inference. This solution can help you go from model creation to production-ready predictions quickly, efficiently, and without managing any infrastructure.
Solution overview
To demonstrate serverless endpoint creation for a SageMaker Canvas trained model, let’s explore an example workflow:

Add the trained model to the Amazon SageMaker Model Registry.
Create a new SageMaker model with the correct configuration.
Create a serverless endpoint configuration.
Deploy the serverless endpoint with the created model and endpoint configuration.

You can also automate the process, as illustrated in the following diagram.

In this example, we deploy a pre-trained regression model to a serverless SageMaker endpoint. This way, we can use our model for variable workloads that don’t require real-time inference.
Prerequisites
As a prerequisite, you must have access to Amazon Simple Storage Service (Amazon S3) and Amazon SageMaker AI. If you don’t already have a SageMaker AI domain configured in your account, you also need permissions to create a SageMaker AI domain.
You must also have a regression or classification model that you have trained. You can train your SageMaker Canvas model as you normally would. This includes creating the Amazon SageMaker Data Wrangler flow, performing necessary data transformations, and choosing the model training configuration. If you don’t already have a trained model, you can follow one of the labs in the Amazon SageMaker Canvas Immersion Day to create one before continuing. For this example, we use a classification model that was trained on the canvas-sample-shipping-logs.csv sample dataset.
Save your model to the SageMaker Model Registry
Complete the following steps to save your model to the SageMaker Model Registry:

On the SageMaker AI console, choose Studio to launch Amazon SageMaker Studio.
In the SageMaker Studio interface, launch SageMaker Canvas, which will open in a new tab.

Locate the model and model version that you want to deploy to your serverless endpoint.
On the options menu (three vertical dots), choose Add to Model Registry.

You can now exit SageMaker Canvas by logging out. To manage costs and prevent additional workspace charges, you can also configure SageMaker Canvas to automatically shut down when idle.
Approve your model for deployment
After you have added your model to the Model Registry, complete the following steps:

In the SageMaker Studio UI, choose Models in the navigation pane.

The model you just exported from SageMaker Canvas should be added with a deployment status of Pending manual approval.

Choose the model version you want to deploy and update the status to Approved by choosing the deployment status.

Choose the model version and navigate to the Deploy tab. This is where you will find the information related to the model and associated container.
Select the container and model location related to the trained model. You can identify it by checking the presence of the environment variable SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT.

Create a new model
Complete the following steps to create a new model:

Without closing the SageMaker Studio tab, open a new tab and open the SageMaker AI console.
Choose Models in the Inference section and choose Create model.
Name your model.
Leave the container input option as Provide model artifacts and inference image location and used the CompressedModel type.
Enter the Amazon Elastic Container Registry (Amazon ECR) URI, Amazon S3 URI, and environment variables that you located in the previous step.

The environment variables will be shown as a single line in SageMaker Studio, with the following format:

SAGEMAKER_DEFAULT_INVOCATIONS_ACCEPT: text/csv, SAGEMAKER_INFERENCE_OUTPUT: predicted_label, SAGEMAKER_INFERENCE_SUPPORTED: predicted_label, SAGEMAKER_PROGRAM: tabular_serve.py, SAGEMAKER_SUBMIT_DIRECTORY: /opt/ml/model/code

You might have different variables than those in the preceding example. All variables from your environment variables should be added to your model. Make sure that each environment variable is on its own line when creating you new model.

Choose Create model.

Create an endpoint configuration
Complete the following steps to create an endpoint configuration:

On the SageMaker AI console, choose Endpoint configurations to create a new model endpoint configuration.
Set the type of endpoint to Serverless and set the model variant to the model created in the previous step.

Choose Create endpoint configuration.

Create an endpoint
Complete the following steps to create an endpoint:

On the SageMaker AI console, choose Endpoints in the navigation pane and create a new endpoint.
Name the endpoint.
Select the endpoint configuration created in the previous step and choose Select endpoint configuration.
Choose Create endpoint.

The endpoint might take a few minutes to be created. When the status is updated to InService, you can begin calling the endpoint.
The following sample code demonstrates how you can call an endpoint from a Jupyter notebook located in your SageMaker Studio environment:

import boto3
import csv
from io import StringIO
import time

def invoke_shipping_prediction(features):
sagemaker_client = boto3.client(‘sagemaker-runtime’)

# Convert to CSV string format
output = StringIO()
csv.writer(output).writerow(features)
payload = output.getvalue()

response = sagemaker_client.invoke_endpoint(
EndpointName=’canvas-shipping-data-model-1-serverless-endpoint’,
ContentType=’text/csv’,
Accept=’text/csv’,
Body=payload
)

response_body = response[‘Body’].read().decode()
reader = csv.reader(StringIO(response_body))
result = list(reader)[0] # Get first row

# Parse the response into a more usable format
prediction = {
‘predicted_label’: result[0],
‘confidence’: float(result[1]),
‘class_probabilities’: eval(result[2]),
‘possible_labels’: eval(result[3])
}

return prediction

# Features for inference
features_set_1 = [
“Bell”,
“Base”,
14,
6,
11,
11,
“GlobalFreight”,
“Bulk Order”,
“Atlanta”,
“2020-09-11 00:00:00”,
“Express”,
109.25199890136719
]

features_set_2 = [
“Bell”,
“Base”,
14,
6,
15,
15,
“MicroCarrier”,
“Single Order”,
“Seattle”,
“2021-06-22 00:00:00”,
“Standard”,
155.0483856201172
]

# Invoke the SageMaker endpoint for feature set 1
start_time = time.time()
result = invoke_shipping_prediction(features_set_1)

# Print Output and Timing
end_time = time.time()
total_time = end_time – start_time

print(f”Total response time with endpoint cold start: {total_time:.3f} seconds”)
print(f”Prediction for feature set 1: {result[‘predicted_label’]}”)
print(f”Confidence for feature set 1: {result[‘confidence’]*100:.2f}%”)
print(“nProbabilities for feature set 1:”)
for label, prob in zip(result[‘possible_labels’], result[‘class_probabilities’]):
print(f”{label}: {prob*100:.2f}%”)

print(“———————————————————“)

# Invoke the SageMaker endpoint for feature set 2
start_time = time.time()
result = invoke_shipping_prediction(features_set_2)

# Print Output and Timing
end_time = time.time()
total_time = end_time – start_time

print(f”Total response time with warm endpoint: {total_time:.3f} seconds”)
print(f”Prediction for feature set 2: {result[‘predicted_label’]}”)
print(f”Confidence for feature set 2: {result[‘confidence’]*100:.2f}%”)
print(“nProbabilities for feature set 2:”)
for label, prob in zip(result[‘possible_labels’], result[‘class_probabilities’]):
print(f”{label}: {prob*100:.2f}%”)

Automate the process
To automatically create serverless endpoints each time a new model is approved, you can use the following YAML file with AWS CloudFormation. This file will automate the creation of SageMaker endpoints with the configuration you specify.
This sample CloudFormation template is provided solely for inspirational purposes and is not intended for direct production use. Developers should thoroughly test this template according to their organization’s security guidelines before deployment.

AWSTemplateFormatVersion: “2010-09-09”
Description: Template for creating Lambda function to handle SageMaker model
package state changes and create serverless endpoints

Parameters:
MemorySizeInMB:
Type: Number
Default: 1024
Description: Memory size in MB for the serverless endpoint (between 1024 and 6144)
MinValue: 1024
MaxValue: 6144

MaxConcurrency:
Type: Number
Default: 20
Description: Maximum number of concurrent invocations for the serverless endpoint
MinValue: 1
MaxValue: 200

AllowedRegion:
Type: String
Default: “us-east-1”
Description: AWS region where SageMaker resources can be created

AllowedDomainId:
Type: String
Description: SageMaker Studio domain ID that can trigger deployments
NoEcho: true

AllowedDomainIdParameterName:
Type: String
Default: “/sagemaker/serverless-deployment/allowed-domain-id”
Description: SSM Parameter name containing the SageMaker Studio domain ID that can trigger deployments

Resources:
AllowedDomainIdParameter:
Type: AWS::SSM::Parameter
Properties:
Name: !Ref AllowedDomainIdParameterName
Type: String
Value: !Ref AllowedDomainId
Description: SageMaker Studio domain ID that can trigger deployments

SageMakerAccessPolicy:
Type: AWS::IAM::ManagedPolicy
Properties:
Description: Managed policy for SageMaker serverless endpoint creation
PolicyDocument:
Version: “2012-10-17”
Statement:
– Effect: Allow
Action:
– sagemaker:CreateModel
– sagemaker:CreateEndpointConfig
– sagemaker:CreateEndpoint
– sagemaker:DescribeModel
– sagemaker:DescribeEndpointConfig
– sagemaker:DescribeEndpoint
– sagemaker:DeleteModel
– sagemaker:DeleteEndpointConfig
– sagemaker:DeleteEndpoint
Resource: !Sub “arn:aws:sagemaker:${AllowedRegion}:${AWS::AccountId}:*”
– Effect: Allow
Action:
– sagemaker:DescribeModelPackage
Resource: !Sub “arn:aws:sagemaker:${AllowedRegion}:${AWS::AccountId}:model-package/*/*”
– Effect: Allow
Action:
– iam:PassRole
Resource: !Sub “arn:aws:iam::${AWS::AccountId}:role/service-role/AmazonSageMaker-ExecutionRole-*”
Condition:
StringEquals:
“iam:PassedToService”: “sagemaker.amazonaws.com”
– Effect: Allow
Action:
– ssm:GetParameter
Resource: !Sub “arn:aws:ssm:${AllowedRegion}:${AWS::AccountId}:parameter${AllowedDomainIdParameterName}”

LambdaExecutionRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument:
Version: “2012-10-17″
Statement:
– Effect: Allow
Principal:
Service: lambda.amazonaws.com
Action: sts:AssumeRole
ManagedPolicyArns:
– arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
– !Ref SageMakerAccessPolicy

ModelDeploymentFunction:
Type: AWS::Lambda::Function
Properties:
Handler: index.handler
Role: !GetAtt LambdaExecutionRole.Arn
Code:
ZipFile: |
import os
import json
import boto3

sagemaker_client = boto3.client(‘sagemaker’)
ssm_client = boto3.client(‘ssm’)

def handler(event, context):
print(f”Received event: {json.dumps(event, indent=2)}”)
try:
# Get details directly from the event
detail = event[‘detail’]
print(f’detail: {detail}’)

# Get allowed domain ID from SSM Parameter Store
parameter_name = os.environ.get(‘ALLOWED_DOMAIN_ID_PARAMETER_NAME’)
try:
response = ssm_client.get_parameter(Name=parameter_name)
allowed_domain = response[‘Parameter’][‘Value’]
except Exception as e:
print(f”Error retrieving parameter {parameter_name}: {str(e)}”)
allowed_domain = ‘*’ # Default fallback

# Check if domain ID is allowed
if allowed_domain != ‘*’:
created_by_domain = detail.get(‘CreatedBy’, {}).get(‘DomainId’)
if created_by_domain != allowed_domain:
print(f”Domain {created_by_domain} not allowed. Allowed: {allowed_domain}”)
return {‘statusCode’: 403, ‘body’: ‘Domain not authorized’}

# Get the model package ARN from the event resources
model_package_arn = event[‘resources’][0]

# Get the model package details from SageMaker
model_package_response = sagemaker_client.describe_model_package(
ModelPackageName=model_package_arn
)

# Parse model name and version from ModelPackageName
model_name, version = detail[‘ModelPackageName’].split(‘/’)
serverless_model_name = f”{model_name}-{version}-serverless”

# Get all container details directly from the event
container_defs = detail[‘InferenceSpecification’][‘Containers’]

# Get the execution role from the event and convert to proper IAM role ARN format
assumed_role_arn = detail[‘CreatedBy’][‘IamIdentity’][‘Arn’]
execution_role_arn = assumed_role_arn.replace(‘:sts:’, ‘:iam:’)
.replace(‘assumed-role’, ‘role/service-role’)
.rsplit(‘/’, 1)[0]

# Prepare containers configuration for the model
containers = []
for i, container_def in enumerate(container_defs):
# Get environment variables from the model package for this container
environment_vars = model_package_response[‘InferenceSpecification’][‘Containers’][i].get(‘Environment’, {}) or {}

containers.append({
‘Image’: container_def[‘Image’],
‘ModelDataUrl’: container_def[‘ModelDataUrl’],
‘Environment’: environment_vars
})

# Create model with all containers
if len(containers) == 1:
# Use PrimaryContainer if there’s only one container
create_model_response = sagemaker_client.create_model(
ModelName=serverless_model_name,
PrimaryContainer=containers[0],
ExecutionRoleArn=execution_role_arn
)
else:
# Use Containers parameter for multiple containers
create_model_response = sagemaker_client.create_model(
ModelName=serverless_model_name,
Containers=containers,
ExecutionRoleArn=execution_role_arn
)

# Create endpoint config
endpoint_config_name = f”{serverless_model_name}-config”
create_endpoint_config_response = sagemaker_client.create_endpoint_config(
EndpointConfigName=endpoint_config_name,
ProductionVariants=[{
‘VariantName’: ‘AllTraffic’,
‘ModelName’: serverless_model_name,
‘ServerlessConfig’: {
‘MemorySizeInMB’: int(os.environ.get(‘MEMORY_SIZE_IN_MB’)),
‘MaxConcurrency’: int(os.environ.get(‘MAX_CONCURRENT_INVOCATIONS’))
}
}]
)

# Create endpoint
endpoint_name = f”{serverless_model_name}-endpoint”
create_endpoint_response = sagemaker_client.create_endpoint(
EndpointName=endpoint_name,
EndpointConfigName=endpoint_config_name
)

return {
‘statusCode’: 200,
‘body’: json.dumps({
‘message’: ‘Serverless endpoint deployment initiated’,
‘endpointName’: endpoint_name
})
}

except Exception as e:
print(f”Error: {str(e)}”)
raise
Runtime: python3.12
Timeout: 300
MemorySize: 128
Environment:
Variables:
MEMORY_SIZE_IN_MB: !Ref MemorySizeInMB
MAX_CONCURRENT_INVOCATIONS: !Ref MaxConcurrency
ALLOWED_DOMAIN_ID_PARAMETER_NAME: !Ref AllowedDomainIdParameterName

EventRule:
Type: AWS::Events::Rule
Properties:
Description: Rule to trigger Lambda when SageMaker Model Package state changes
EventPattern:
source:
– aws.sagemaker
detail-type:
– SageMaker Model Package State Change
detail:
ModelApprovalStatus:
– Approved
UpdatedModelPackageFields:
– ModelApprovalStatus
State: ENABLED
Targets:
– Arn: !GetAtt ModelDeploymentFunction.Arn
Id: ModelDeploymentFunction

LambdaInvokePermission:
Type: AWS::Lambda::Permission
Properties:
FunctionName: !Ref ModelDeploymentFunction
Action: lambda:InvokeFunction
Principal: events.amazonaws.com
SourceArn: !GetAtt EventRule.Arn

Outputs:
LambdaFunctionArn:
Description: ARN of the Lambda function
Value: !GetAtt ModelDeploymentFunction.Arn
EventRuleArn:
Description: ARN of the EventBridge rule
Value: !GetAtt EventRule.Arn

This stack will limit automated serverless endpoint creation to a specific AWS Region and domain. You can find your domain ID when accessing SageMaker Studio from the SageMaker AI console, or by running the following command: aws sagemaker list-domains —region [your-region]
Clean up
To manage costs and prevent additional workspace charges, make sure that you have logged out of SageMaker Canvas. If you tested your endpoint using a Jupyter notebook, you can shut down your JupyterLab instance by choosing Stop or configuring automated shutdown for JupyterLab.

In this post, we showed how to deploy a SageMaker Canvas model to a serverless endpoint using SageMaker Serverless Inference. By using this serverless approach, you can quickly and efficiently serve predictions from your SageMaker Canvas models without needing to manage the underlying infrastructure.
This seamless deployment experience is just one example of how AWS services like SageMaker Canvas and SageMaker Serverless Inference simplify the ML journey, helping businesses of different sizes and technical proficiencies unlock the value of AI and ML. As you continue exploring the SageMaker ecosystem, be sure to check out how you can unlock data governance for no-code ML with Amazon DataZone, and seamlessly transition between no-code and code-first model development using SageMaker Canvas and SageMaker Studio.

About the authors
Nadhya Polanco is a Solutions Architect at AWS based in Brussels, Belgium. In this role, she supports organizations looking to incorporate AI and Machine Learning into their workloads. In her free time, Nadhya enjoys indulging in her passion for coffee and traveling.
Brajendra Singh is a Principal Solutions Architect at Amazon Web Services, where he partners with enterprise customers to design and implement innovative solutions. With a strong background in software development, he brings deep expertise in Data Analytics, Machine Learning, and Generative AI.

Amazon Nova Sonic is a foundation model that creates natural, human-like speech-to-speech conversations for generative AI applications, allowing users to interact with AI through voice in real-time, with capabilities for understanding tone, enabling natural flow, and performing actions.
Multi-agent architecture offers a modular, robust, and scalable design pattern for production-level voice assistants. This blog post explores Amazon Nova Sonic voice agent applications and demonstrates how they integrate with Strands Agents framework sub-agents while leveraging Amazon Bedrock AgentCore to create an effective multi-agent system.
Why multi-agent architecture?
Imagine developing a financial assistant application responsible for user onboarding, information collection, identity verification, account inquiries, exception handling, and handing off to human agents based on predefined conditions. As functional requirements expand, the voice agent continues to add new inquiry types. The system prompt grows enormous, and the underlying logic becomes increasingly complex, illustrates a persistent challenge in software development: monolithic designs lead to systems that are difficult to maintain and enhance.
Think of multi-agent architecture as building a team of specialized AI assistants rather than relying on a single do-it-all helper. Just like companies divide responsibilities across different departments, this approach breaks complex tasks into smaller, manageable pieces. Each AI agent becomes an expert in a specific area—whether that’s fact-checking, data processing, or handling specialized requests. For the user, the experience feels seamless: there’s no delay, no change in voice, and no visible handoff. The system functions behind the scenes, directing each expert agent to step in at the right moment.
In addition to modular and robust benefits, multi-agent systems offer advantages similar to a microservice architecture, a popular enterprise software design pattern, providing scalability, distribution and maintainability while allowing organizations to reuse agentic workflows already developed for their large language model (LLM)-powered applications.
Sample application
In this blog, we refer to the Amazon Nova Sonic workshop multi-agent lab code, which uses the banking voice assistant as a sample to demonstrate how to deploy specialized agents on Amazon Bedrock AgentCore. It uses Nova Sonic as the voice interface layer and acts as an orchestrator to delegate detailed inquiries to sub-agents written in Strands Agents hosted on AgentCore Runtime. You can find the sample source code on the GitHub repo.
In the banking voice agent sample, the conversation flow begins with a greeting and collecting the user’s name, and then it handles inquiries related to banking or mortgages. We use three secondary level agents hosted on AgentCore to handle specialized logic:

Authenticate sub-agent: Handles user authentication using the account ID and other information
Banking sub-agent: Handles account balance checks, statements, and other banking-related inquiries
Mortgage sub-agent: Handles mortgage-related inquiries, including refinancing, rates, and repayment options

Sub-agents are self-contained, handling their own logic such as input validation. For instance, the authentication agent validates account IDs and returns errors to Nova Sonic if needed. This simplifies the reasoning logic in Nova Sonic while keeping business logic encapsulated, similar to the software engineering modular design patterns.
Integrate Nova Sonic with AgentCore through tool use events
Amazon Nova Sonic relies on tool use to integrate with agentic workflows. During the Nova Sonic event lifecycle, you can provide tool use configurations through the promptStart event, which is designed to initiate when Sonic receives specific types of input.
For example, in the following Sonic tool configuration sample, tool use is configured to initiate events based on Sonic’s built-in reasoning model, which classifies the inquiry for routing to the banking sub-agents.

[
    {
        “toolSpec”: {
            “name”: “bankAgent”,
            “description”: `Use this tool whenever the customer asks about their **bank account balance** or **bank statement**.  
                    It should be triggered for queries such as:  
                    – “What’s my balance?”  
                    – “How much money do I have in my account?”  
                    – “Can I see my latest bank statement?”  
                    – “Show me my account summary.”`,
            “inputSchema”: {
                “json”: JSON.stringify({
                “type”: “object”,
                “properties”: {
                    “accountId”: {
                        “type”: “string”,
                        “description”: “This is a user input. It is the bank account Id which is a numeric number.”
                    },
                    “query”: {
                        “type”: “string”,
                        “description”: “The inquiry to the bank agent such as check account balance, get statement etc.”
                    }
                },
                “required”: [
                    “accountId”, “query”
                ]
                })
            }
        }
    }
]

When a user asks Nova Sonic a question such as ‘What is my account balance?’, Sonic sends a toolUse event to the client application with the specified toolName (for example, bankAgent) defined in the configuration. The application can then invoke the sub-agent hosted on AgentCore to handle the banking logic and return the response to Sonic, which in turn generates an audio reply for the user.

{
“event”: {
“toolUse”: {
“completionId”: “UUID”,
“content”: “{“accountId”:”one two three four five”,”query”:”check account balance”}”,
“contentId”: “UUID”,
“promptName”: “UUID”,
“role”: “TOOL”,
“sessionId”: “UUID”,
“toolName”: “bankAgent”,
“toolUseId”: “UUID”
}
}
}

Sub-agent on AgentCore
The following sample showcases the banking sub-agent developed using the Strands Agents framework, specifically configured for deployment on Bedrock AgentCore. It leverages Nova Lite through Amazon Bedrock as its reasoning model, providing effective cognitive capabilities with minimal latency. The agent implementation features a system prompt that defines its banking assistant responsibilities, complemented by two specialized tools: one for account balance inquiries and another for bank statement retrieval.

from strands import Agent, tool
import json
from bedrock_agentcore.runtime import BedrockAgentCoreApp
from strands.models import BedrockModel
import re, argparse

app = BedrockAgentCoreApp()

@tool
def get_account_balance(account_id) -> str:
    “””Get account balance for given account Id

    Args:
        account_id: Bank account Id
    “””

    # The actual implementation will retrieve information from a database API or another backend service.
    
    return {“result”: result}

@tool
def get_statement(account_id: str, year_and_month: str) -> str:
    “””Get account statement for a given year and month
    Args:
        account_id: Bank account Id
        year_and_month: Year and month of the bank statement. For example: 2025_08 or August 2025
    “””
    # The actual implementation will retrieve information from a database API or another backend service.
    
    return {“result”: result}

# Specify Bedrock LLM for the Agent
bedrock_model = BedrockModel(
    model_id=”amazon.nova-lite-v1:0″,
)
# System prompt
system_prompt = ”’
You are a banking agent. You will receive requests that include:  
– `account_id`  
– `query` (the inquiry type, such as **balance** or **statement**, plus any additional details like month).  

## Instructions
1. Use the provided `account_id` and `query` to call the tools.  
2. The tool will return a JSON response.  
3. Summarize the result in 2–3 sentences.  
   – For a **balance inquiry**, give the account balance with currency and date.  
   – For a **statement inquiry**, provide opening balance, closing balance, and number of transactions.  
4. Do not return raw JSON. Always respond in natural language.  
”’

# Create an agent with tools, LLM, and system prompt
agent = Agent(
    tools=[ get_account_balance, get_statement],
    model=bedrock_model,
    system_prompt=system_prompt
)

@app.entrypoint
def banking_agent(payload):
    response = agent(json.dumps(payload))
    return response.message[‘content’][0][‘text’]
    
if __name__ == “__main__”:
    app.run()

Best practices for voice-based multi-agent systems
Multi-agent architecture provides exceptional flexibility and a modular design approach, allowing developers to structure voice assistants efficiently and potentially reuse existing specialized agent workflows. When implementing voice-first experiences, there are important best practices to consider that address the unique challenges of this modality.

Balance flexibility and latency: Although the ability to invoke sub-agents using Nova Sonic tool use events creates powerful capabilities, it can introduce additional latency to voice responses. For the use cases that require a synchronized experience, each agent handoff represents a potential delay point in the interaction flow. Therefore, it’s important to design with response time in mind.
Optimize model selection for sub-agents: Starting with smaller, more efficient models like Nova Lite for sub-agents can significantly reduce latency while still handling specialized tasks effectively. Reserve larger, more capable models for complex reasoning or when sophisticated natural language understanding is essential.
Craft voice-optimized responses: Voice assistants perform best with concise, focused responses that can be followed by additional details when needed. This approach not only improves latency but also creates a more natural conversational flow that aligns with human expectations for verbal communication.

Consider stateless vs. stateful sub-agent design
Stateless sub-agents handle each request independently, without retaining memory of past interactions or session-level states. They are simple to implement, easy to scale, and work well for straightforward, one-off tasks. However, they cannot provide context-aware responses unless external state management is introduced.
Stateful sub-agents, on the other hand, maintain memory across interactions to support context-aware responses and session-level states. This enables more personalized and cohesive user experiences, but comes with added complexity and resource requirements. They are best suited for scenarios involving multi-turn interactions and user or session-level context caching.
Conclusion
Multi-agent architectures unlock flexibility, scalability, and accuracy for complex AI-driven workflows. By combining the Nova Sonic conversational capabilities with the orchestration power of Bedrock AgentCore, you can build intelligent, specialized agents that work together seamlessly. If you’re exploring ways to enhance your AI applications, multi-agent patterns with Nova Sonic and AgentCore are a powerful approach worth testing.
Learn more about Amazon Nova Sonic by visiting the User Guide, building your application with the sample applications, and exploring the Nova Sonic workshop to get started. You can also refer to the technical report and model card for additional benchmarks.

About the authors
Lana Zhang is a Senior Specialist Solutions Architect for Generative AI at AWS within the Worldwide Specialist Organization. She specializes in AI/ML, with a focus on use cases such as AI voice assistants and multimodal understanding. She works closely with customers across diverse industries, including media and entertainment, gaming, sports, advertising, financial services, and healthcare, to help them transform their business solutions through AI.

Large-scale AI model training faces significant challenges with failure recovery and monitoring. Traditional training requires complete job restarts when even a single training process fails, resulting in additional downtime and increased costs. As training clusters expand, identifying and resolving critical issues like stalled GPUs and numerical instabilities typically requires complex custom monitoring code.
With Amazon SageMaker HyperPod you can accelerate AI model development across hundreds or thousands of GPUs with built-in resiliency, decreasing model training time by up to 40%. The Amazon SageMaker HyperPod training operator further enhances training resilience for Kubernetes workloads through pinpoint recovery and customizable monitoring capabilities.
In this blog post, we show you how to deploy and manage machine learning training workloads using the Amazon SageMaker HyperPod training operator, including setup instructions and a complete training example.
Amazon SageMaker HyperPod training operator
The Amazon SageMaker HyperPod training operator helps you accelerate generative AI model development by efficiently managing distributed training across large GPU clusters. The Amazon SageMaker HyperPod training operator uses built-in fault resiliency components, comes packaged as an Amazon Elastic Kubernetes Service (Amazon EKS) add-on, and deploys the necessary custom resource definitions (CRDs) to the HyperPod cluster.
Solution overview
The following diagram depicts the architecture of Amazon SageMaker HyperPod training operator.

The HyperPod training operator follows Kubernetes operator pattern and has the following major components:

Custom Resource Definition (CRDs): HyperPodPyTorchJob defines the job specification (for example, node count, image) and serves as the interface for customers to submit jobs. apiVersion: sagemaker.amazonaws.com/v1 kind: HyperPodPyTorchJob
RBAC policies: Defines the actions the controller is allowed to perform, such as creating pods and managing HyperPodPyTorchJob resources.
Job controller: Listens to job creation and fulfills requests by creating job pods and pod managers.
Pod manager: Monitors training process health on each pod. The number of Pod Managers is determined by the number of pods required by the job. One Pod Manager currently controls several hundred pods.
HyperPod elastic agent: Customers install the elastic agent into their training container. It orchestrates lifecycles of training workers on each container and communicates with the Amazon SageMaker HyperPod training operator. The HyperPod elastic agent is an extension of PyTorch’s ElasticAgent.

The job Controller uses fault detection components such as the SageMaker HyperPod health-monitoring agent and node health check mechanisms like AWS retirement notices to update job state and repair faults. It also relies on the HyperPod elastic agent to check the status of training processes for crashes and hung job detection.
When a HyperPodPyTorch job is submitted, the Amazon SageMaker HyperPod training operator spins up job pods along with pod manager pods that help manage the training job lifecycle. The pod managers interact with the HyperPod elastic agent so that all job pods maintain a healthy state.
Benefits of using the operator
The Amazon SageMaker HyperPod training operator can be installed as an EKS add-on on your cluster. The key benefits include:

Centralized training process monitoring and restart – The HyperPod training operator maintains a control plane with a global view of health across all ranks. When one rank encounters an issue, it broadcasts a stop signal to all ranks to prevent other ranks from failing individually at different times due to collective communication timeout. This supports more efficient fault detection and recovery.
Centralized efficient rank assignment – A separate HyperPod rendezvous backend allows the HyperPod training operator to assign ranks directly. This reduces initialization overhead by eliminating the need for worker-to-worker discovery.
Unhealthy training node detection and job restart – The HyperPod training operator is fully integrated with the HyperPod EKS cluster resiliency features, helping restart jobs or training processes due to bad nodes and hardware issues in ML workloads. This reduces the need to self-manage job recovery solutions.
Granular process recovery – Rather than restarting entire jobs when failures occur, the operator precisely targets and restarts only training processes, reducing recovery times from tens of minutes to seconds. This makes HyperPod training operator job recovery time scale linearly as cluster size grows.
Hanging job detection and performance degradation detection – Based on training script log monitoring, the HyperPod training operator helps overcome problematic training scenarios including stalled training batches, non-numeric loss values, and performance degradation through simple YAML configurations. For more information see, Using the training operator to run jobs in the Amazon SageMaker AI Developer Guide.

Training operator setup
This section walks through installing the Amazon SageMaker HyperPod training operator as an Amazon EKS add-on.
Estimated Setup Time: 30-45 minutes
Prerequisites
Before getting started, verify that you have the following resources and permissions.
Required AWS resources:

Active AWS account
Amazon EKS cluster (version 1.28 or later)
Amazon SageMaker HyperPod EKS cluster
Amazon ECR repository for container images

Required IAM permissions:

AmazonSageMakerHyperPodTrainingOperatorAccess managed policy
EKS cluster access permissions
ECR push/pull permissions
eks-pod-identity-agent add-on installed on EKS cluster

Required software:

kubectl (version 1.28 or later), for more information see the kubectl installation documentation
docker (version 20.10 or later), for more information see the docker installation documentation
AWS Command Line Interface (AWS CLI) (version 2.0 or later), for more information see the AWS CLI installation documentation
envsubst utility
HuggingFace account with access token

Installation instructions
Before running the installation steps below, you’ll need to first create a HyperPod cluster. If you haven’t done this one already please follow the instructions to create an EKS-orchestrated SageMaker HyperPod cluster to get started. Make sure to install eks-pod-identity-agent add-on on the EKS cluster, by following the Set up the Amazon EKS Pod Identity Agent instructions.
Install cert-manager
First, install the cert-manager add-on which is required for the HyperPod training operator:

Open the Amazon EKS console
Navigate to your EKS cluster and go to the Add-ons page
On the Add-ons page, locate Get more add-ons and navigate to the Community add-ons section
Find the Cert Manager add-on, select it, and choose Next
On the add-on configuration page, proceed with default settings and choose Next
Preview all selections for the Cert Manager add-on and choose Create
Wait for the add-on status to change to Active before proceeding

Install the HyperPod training operator add-on
Once cert-manager is active, install the Amazon SageMaker HyperPod training operator:

Open the Amazon SageMaker console
Navigate to your cluster’s details page
On the Dashboard tab, locate Amazon SageMaker HyperPod training operator and choose Install

During installation, SageMaker creates an IAM execution role with permissions similar to the AmazonSageMakerHyperPodTrainingOperatorAccess managed policy and creates a pod identity association between your Amazon EKS cluster and the new execution role.

Verify installation
We have now successfully setup of the Amazon SageMaker HyperPod training operator. You can confirm that the pods are running by using the following command:

kubectl -n aws-hyperpod get pods -l hp-training-control-plane=hp-training-operator-controller-manager

Your output should contain the training operator controller as shown below:

NAME READY      STATUS         RESTARTS        AGE
hp-training-operator-hp-training-controller-manager-85c68bmd79b    1/1              Running         0                        24m

Set up training job
Let’s run a PyTorch-based training example on a Llama model. We begin by checking out the following code base:

git clone 
cd awsome-distributed-training/tree/main/3.test_cases/pytorch/FSDP

These scripts provide an easy way to get started with multinode FSDP training on EKS. It is designed to be as simple as possible, requires no data preparation, and uses a container image.
Next, build the docker container image.

aws ecr-public get-login-password —region us-east-1 | docker login —username AWS —password-stdin public.ecr.aws/hpc-cloud
export REGION=$(aws ec2 describe-availability-zones —output text —query ‘AvailabilityZones[0].[RegionName]’)
export ACCOUNT=$(aws sts get-caller-identity —query Account —output text)
export REGISTRY=${ACCOUNT}.dkr.ecr.${REGION}.

docker build -t ${REGISTRY}fsdp:pytorch2.5.1 .    

The above command works with linux based environments, if you are on a Mac, use buildx to target linux/amd64 architecture:

docker buildx build –platform linux/amd64 -t ${REGISTRY}fsdp:pytorch2.5.1 .

Push the image to Amazon ECR:

# Create registry if needed
REGISTRY_COUNT=$(aws ecr describe-repositories | grep “fsdp” | wc -l)
if [ “$REGISTRY_COUNT” -eq 0 ]; then
    aws ecr create-repository –repository-name fsdp
fi

# Login to registry
echo “Logging in to $REGISTRY …”
aws ecr get-login-password | docker login –username AWS –password-stdin $REGISTRY

# Push image to registry
docker image push ${REGISTRY}fsdp:pytorch2.5.1

Note: Pushing the image may take some time depending on your network bandwidth.
Data
For this example, we’ll be using the allenai/c4 dataset. Instead of downloading the whole thing, the create_streaming_dataloaders function will stream the dataset from HuggingFace, so there’s no data prep required for running this training.
If you’d like to instead use your own dataset, you can do so by formatting it as a HuggingFace dataset, and passing its location to the –dataset_path argument.
For the dataset, you will need a Hugging Face access token. First, create a Hugging Face account. Then generate your access token with read permissions.
We will reference this token in the next step by setting it as an environment variable.
This example uses envsubst to generate a Kubernetes manifest file from a template file and parameters. If you don’t have envsubst on your development environment, install it by following the installation instructions.
Launch Llama 3.1 8B training job
Next, we generate the Kubernetes manifest and apply it to the cluster. Let’s navigate to the FSDP source repo:

cd awsome-distributed-training/tree/main/3.test_cases/pytorch/FSDP/Kubernetes

Here, we start by creating environment variables that are used in our training job. Fill out the placeholders as per your cluster size.

cat << EOF > env_vars
export ACCOUNT_ID=<AWS_ACCOUNT_ID>
export REGION=<REGION>
export REGISTRY=${ACCOUNT_ID}.dkr.ecr.${REGION}.amazonaws.com
export IMAGE_URI=${REGISTRY}/fsdp:pytorch2.5.1
export INSTANCE_TYPE=<INSTANCE TYPE> # ml.p5.48xlarge
export NUM_NODES=<NUMBER OF NODES> # 2
export GPU_PER_NODE=<NUMBER OF GPUS PER NODE> # 8
export EFA_PER_NODE=<NUMBER OF EFA PER NODE> # 32
export FI_PROVIDER=efa
export HF_TOKEN=<YOUR HF ACCESS TOKEN> # HF_xxxx
EOF

Once you fill in env_vars and then source variables:

source env_vars

You can apply yaml to submit the training job:

envsubst < llama3_1_8b-fsdp-hpto.yaml | kubectl apply -f –

You can also adjust the training parameters in the TRAINING_ARGS section of the llama3_1_8b-fsdp-hpto.yaml. Additional parameters can be found under model/arguments.py. Note that we use the same directory for both –checkpoint_dir and –resume_from_checkpoint. If there are multiple checkpoints, –resume_from_checkpoint will automatically select the most recent one. This way if our training is interrupted for any reason, it will automatically pick up the most recent checkpoint.
Additionally, you can also prepare and submit your jobs compatible with the Amazon SageMaker HyperPod training operator through the HyperPod CLI and SDK capabilities that have been recently announced, more reading information on how to use it is available in this development guide.
Monitor training job
To see the status of your job, use the following command:

kubectl get hyperpodpytorchjobs

Use the following command to list the jobs ran using HyperPod training operator:

NAME              AGE
llama2-13b-fsdp   2m15s

kubectl get pods 

Use the following command to list all the pods for the training jobs:

NAME                    READY  STATUS   RESTARTS AGE
llama2-13b-fsdp-pods-0  1/1    Running   0       13s
llama2-13b-fsdp-pods-1  1/1    Running   0       13s
llama2-13b-fsdp-pods-2  1/1    Running   0       13s
llama2-13b-fsdp-pods-3  1/1    Running   0       13s

To check the pod logs run the below command to continuously stream the logs to stdout, use the following command:

kubectl logs -f llama2-13b-fsdp-pods-0

Configure log monitoring
With Amazon SageMaker HyperPod training operators users can configure log patterns that the operator continuously monitors. The HyperPod operator continuously looks for the configured regex pattern and stops the training job if it finds a violation. The llama3_1_13b-fsdp-hpto.yaml file that we used previously contains log monitoring configurations for tracking Job start hangs, hang detection during training, and checkpoint creation failures as shown below:

 logMonitoringConfiguration:
      – name: “JobStart”
        logPattern: “.*Loss:.*”
        expectedStartCutOffInSeconds: 240 # job should print loss within 4 mins of start time
      – name: “JobHangingDetection”
        logPattern: “.*Loss:.*”
        expectedRecurringFrequencyInSeconds: 300 # if next batch is not printed within 300 seconds
      – name: “NoSCheckpointingDetection”
        logPattern: “.*Completed checkpoint.*”
        expectedRecurringFrequencyInSeconds: 600 # If next checkpoint upload doesn’t happen within 10 mins, mark it hang.
        expectedStartCutOffInSeconds: 900 # Allow 30 minutes for first checkpoint upload

And the corresponding code files in /src/train.py have the necessary log statements.

logger.info(
             “Batch %d Loss: %.5f, Speed: %.2f samples/sec, lr: %.6f”,  # pylint: disable=line-too-long
                    batch_idx,
                    loss_scalar,
                    throughput,
                    current_lr,
                )
               

Any time these metrics exhibit deviation from their expected values, the operator will detect it as a fault, and trigger a recovery process to re-execute the job, up to a user-specified maximum number of retries.
Additionally, the HyperPod training operator also supports integration with Amazon SageMaker Task Governance.
Integration with HyperPod Observability
SageMaker HyperPod offers a managed observability experience through the newly launched the HyperPod Monitoring and Observability EKS add-on. The observability add-on automatically populates Kubeflow Training metrics in Grafana dashboards out of the box, but for HyperPod PyTorch job metrics, you would have to turn on the advanced training metrics which leverage the HyperPod training operator to show information around job downtime, job recovery and faults, and downtime.
To get these advanced metrics, you can refer to Setting up the SageMaker HyperPod observability add-on. This helps to streamline the process of manually setting up a scraper and building dashboards.
Clean up
To avoid incurring unnecessary charges, clean up the resources created in this walkthrough.
Delete training jobs
Remove all HyperPod training jobs:

kubectl delete hyperpodpytorchjobs –all

Verify jobs are deleted:

kubectl get hyperpodpytorchjobs

Remove container images
Delete the ECR repository and images:

aws ecr delete-repository –repository-name fsdp –force

Remove add-ons:
Remove the following add-ons:
Remove the Amazon SageMaker HyperPod training operator add-on:

Open the Amazon SageMaker console
Navigate to your cluster’s details page
On the Add-ons tab, select the Amazon SageMaker HyperPod training operator
Choose Remove

Remove the cert manager add-on:

Open the Amazon EKS console
Navigate to your EKS cluster’s Add-ons page
Select Cert Manager and choose Remove

Additional clean up
Consider removing these resources if no longer needed:

Any persistent volumes created during training
CloudWatch log groups (if you want to retain logs, leave these)
Custom IAM roles created specifically for this example
The HyperPod cluster itself (if no longer needed).

Conclusion
As organizations continue to push the boundaries of AI model development, tools like the Amazon SageMaker HyperPod training operator can be used to maintain efficiency and reliability at scale. Amazon SageMaker HyperPod training operator offers a robust solution to common challenges in large model training. Key takeaways include:

One-click installation through AWS SageMaker HyperPod cluster console user-interface.
Custom rendezvous backend eliminates initialization and worker synchronization overhead which results in faster job starts and recovery.
Process level restarts maximize recovery efficiency when runtime faults occur.
Customizable hang job detection during training.
Comprehensive monitoring for early detection of training issues.
Out-of-box integration with existing HyperPod resiliency features.

To get started with the Amazon SageMaker HyperPod training operator, follow the setup instructions provided in this post and explore the example training job to understand how it can benefit your specific use case. For more information and best practices, visit the Amazon SageMaker documentation.

About the authors
Arun Kumar Lokanatha is a Senior ML Solutions Architect with the Amazon SageMaker AI. He holds a Master’s degree from UIUC with a specialization in Data science. He specializes in Generative AI workloads, helping customers build and deploy LLM’s using SageMaker HyperPod, SageMaker training jobs, and SageMaker distributed training. Outside of work, he enjoys running, hiking, and cooking.
Haard Mehta is a Software Engineer with Amazon’s SageMaker AI team and holds a Master’s degree in Computer Science with a specialization in big data systems from Arizona State University. He has extensive experience building managed machine learning services at scale, with a focus on hardware resiliency and enabling customers to succeed in their AI use cases without complex infrastructure management. Haard enjoys exploring new places, photography, cooking, and road trips.
Anirudh Viswanathan is a Sr Product Manager, Technical – External Services with the SageMaker AI Training team. He holds a Masters in Robotics from Carnegie Mellon University, an MBA from the Wharton School of Business, and is named inventor on over 40 patents. He enjoys long-distance running, visiting art galleries, and Broadway shows.

A team of researchers from Google Research and UC Santa Cruz released DeepSomatic, an AI model that identifies cancer cell genetic variants. In research with Children’s Mercy, it found 10 variants in pediatric leukemia cells missed by other tools. DeepSomatic has a somatic small variant caller for cancer genomes that works across Illumina short reads, PacBio HiFi long reads, and Oxford Nanopore long reads. The method extends DeepVariant, detects single nucleotide variants and small insertions and deletions in whole genome and whole exome data, and supports tumor normal and tumor only workflows, including FFPE models.

https://research.google/blog/using-ai-to-identify-genetic-variants-in-tumors-with-deepsomatic/?utm_source=twitter&utm_medium=social&utm_campaign=social_post&utm_content=gr-acct

How It Works?

DeepSomatic converts aligned reads into image like tensors that encode pileups, base qualities, and alignment context. A convolutional neural network classifies candidate sites as somatic or not and the pipeline emits VCF or gVCF. This design is platform agnostic because the tensor summarizes local haplotype and error patterns across technologies. Google researchers describe the approach and its focus on distinguishing inherited and acquired variants including difficult samples such as glioblastoma and pediatric leukemia.

Datasets and Benchmarking

Training and evaluation use CASTLE, Cancer Standards Long read Evaluation. CASTLE contains 6 matched tumor and normal cell line pairs that were whole genome sequenced on Illumina, PacBio HiFi, and Oxford Nanopore. The research team releases benchmark sets and accessions for reuse. This fills a gap in multi technology somatic training and testing resources.

https://research.google/blog/using-ai-to-identify-genetic-variants-in-tumors-with-deepsomatic/?utm_source=twitter&utm_medium=social&utm_campaign=social_post&utm_content=gr-acct

Reported Results

The research team report consistent gains over widely used methods in both single nucleotide variants and indels. On Illumina indels, the next best method is about 80 percent F1, DeepSomatic is about 90 percent. On PacBio indels, the next best method is under 50 percent, DeepSomatic is above 80 percent. Baselines include SomaticSniper, MuTect2, and Strelka2 for short reads and ClairS for long reads. The study reports 329,011 somatic variants across the reference lines and an additional preserved sample. Google research team reports that DeepSomatic outperforms current methods with particular strength on indels.

https://research.google/blog/using-ai-to-identify-genetic-variants-in-tumors-with-deepsomatic/?utm_source=twitter&utm_medium=social&utm_campaign=social_post&utm_content=gr-acct

Generalization to Real Samples

The research team evaluates transfer to cancers beyond the training set. A glioblastoma sample shows recovery of known drivers. Pediatric leukemia samples test the tumor only mode where a clean normal is not available. The tool recovers known calls and reports additional variants in that cohort. These studies indicate the representation and training scheme generalize to new disease contexts and to settings without matched normals.

Key Takeaways

DeepSomatic detects somatic SNVs (single nucleotide variants) and indels across Illumina, PacBio HiFi, and Oxford Nanopore, and builds on the DeepVariant methodology.

The pipeline supports tumor normal and tumor only workflows, includes FFPE WGS and WES models, and is released on GitHub.

It encodes read pileups as image like tensors and uses a convolutional neural network to classify somatic sites and emit VCF or gVCF.

Training and evaluation use the CASTLE dataset with 6 matched tumor normal cell line pairs sequenced on three platforms, with benchmarks and accessions provided.

Reported results show about 90 percent indel F1 on Illumina and above 80 percent on PacBio, outperforming common baselines, with 329,011 somatic variants identified across reference samples.

Editorial Comments

DeepSomatic is a pragmatic step for somatic variant calling across sequencing platforms, the model keeps DeepVariant’s image tensor representation and a convolutional neural network, so the same architecture scales from Illumina to PacBio HiFi to Oxford Nanopore with consistent preprocessing and outputs. The CASTLE dataset is the right move, it supplies matched tumor and normal cell lines across 3 technologies, which strengthens training and benchmarking and aids reproducibility. Reported results emphasize indel accuracy, about 90% F1 on Illumina and more than 80% on PacBio against lower baselines, which addresses a long running weakness in indel detection. The pipeline supports WGS and WES, tumor normal and tumor only, and FFPE, which matches real laboratory constraints.

Check out the Technical Paper, Technical details, Dataset and GitHub Repo. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post Google AI Research Releases DeepSomatic: A New AI Model that Identifies Cancer Cell Genetic Variants appeared first on MarkTechPost.

DeepSeek-AI released 3B DeepSeek-OCR, an end to end OCR and document parsing Vision-Language Model (VLM) system that compresses long text into a small set of vision tokens, then decodes those tokens with a language model. The method is simple, images carry compact representations of text, which reduces sequence length for the decoder. The research team reports 97% decoding precision when text tokens are within 10 times the vision tokens on Fox benchmark, and useful behavior even at 20 times compression. It also reports competitive results on OmniDocBench with far fewer tokens than common baselines.

https://github.com/deepseek-ai/DeepSeek-OCR/blob/main/DeepSeek_OCR_paper.pdf

Architecture, what is actually new?

DeepSeek-OCR-3B has two components, a vision encoder named DeepEncoder and a Mixture of Experts decoder named DeepSeek3B-MoE-A570M. The encoder is designed for high resolution inputs with low activation cost and with few output tokens. It uses a window attention stage based on SAM for local perception, a 2 layer convolutional compressor for 16× token downsampling, and a dense global attention stage based on CLIP for visual knowledge aggregation. This design keeps activation memory controlled at high resolution, and keeps the vision token count low. The decoder is a 3B parameter MoE model (named as DeepSeek3B-MoE-A570M) with about 570M active parameters per token.

https://github.com/deepseek-ai/DeepSeek-OCR/blob/main/DeepSeek_OCR_paper.pdf

Multi resolution modes, engineered for token budgets

DeepEncoder supports native modes and dynamic modes. Native modes are Tiny with 64 tokens at 512 by 512 pixels, Small with 100 tokens at 640 by 640, Base with 256 tokens at 1024 by 1024, and Large with 400 tokens at 1280 by 1280. Dynamic modes named Gundam and Gundam-Master mix tiled local views with a global view. Gundam yields n×100 plus 256 tokens, or n×256 plus 400 tokens, with n in the range 2 to 9. For padded modes, the research team gives a formula for valid tokens, which is lower than the raw token count, and depends on the aspect ratio. These modes let AI developers and researchers align token budgets with page complexity.

https://github.com/deepseek-ai/DeepSeek-OCR/blob/main/DeepSeek_OCR_paper.pdf

https://github.com/deepseek-ai/DeepSeek-OCR/blob/main/DeepSeek_OCR_paper.pdf

Compression results, what the numbers say…..

The Fox benchmark study measures precision as exact text match after decoding. With 100 vision tokens, pages with 600 to 700 text tokens reach 98.5% precision at 6.7× compression. Pages with 900 to 1000 text tokens reach 96.8% precision at 9.7× compression. With 64 vision tokens, precision decreases as compression increases, for example 59.1% at about 19.7× for 1200 to 1300 text tokens. These values come directly from Table 2.

https://github.com/deepseek-ai/DeepSeek-OCR/blob/main/DeepSeek_OCR_paper.pdf

On OmniDocBench, the abstract reports that DeepSeek-OCR surpasses GOT-OCR 2.0 when using only 100 vision tokens per page, and that under 800 vision tokens it outperforms MinerU 2.0, which uses over 6000 tokens per page on average. The benchmark section presents overall performance in terms of edit distance.

https://github.com/deepseek-ai/DeepSeek-OCR/blob/main/DeepSeek_OCR_paper.pdf

Training details that matter….

The research team describes a two phase training pipeline. It first trains DeepEncoder with next token prediction on OCR 1.0 and OCR 2.0 data and 100M LAION samples, then trains the full system with pipeline parallelism across 4 partitions. For hardware, the run used 20 nodes, each with 8 A100 40G GPUs, and used AdamW. The team reports a training speed of 90B tokens per day on text only data, and 70B tokens per day on multimodal data. In production, it reports the ability to generate over 200k pages per day on a single A100 40G node.

How to evaluate it in a practical stack

If your target documents are typical reports or books, start with Small mode at 100 tokens, then adjust upward only if the edit distance is unacceptable. If your pages contain dense small fonts or very high token counts, use a Gundam mode, since it combines global and local fields of view with explicit token budgeting. If your workload includes charts, tables, or chemical structures, review the “Deep parsing” qualitative section, which shows conversions to HTML tables and SMILES and structured geometry, then design outputs that are easy to validate.

https://github.com/deepseek-ai/DeepSeek-OCR/blob/main/DeepSeek_OCR_paper.pdf

Key Takeaways

DeepSeek OCR targets token efficiency using optical context compression with near lossless decoding at about 10 times compression, and around 60 percent precision at about 20 times compression.

The HF release expose explicit token budgets, Tiny uses 64 tokens at 512 by 512, Small uses 100 tokens at 640 by 640, Base uses 256 tokens at 1024 by 1024, Large uses 400 tokens at 1280 by 1280, and Gundam composes n views at 640 by 640 plus one global view at 1024 by 1024.

The system structure is a DeepEncoder that compresses pages into vision tokens and a DeepSeek3B MoE decoder with about 570M active parameters, as described by the research team in the technical report.

The Hugging Face model card documents a tested setup for immediate use, Python 3.12.9, CUDA 11.8, PyTorch 2.6.0, Transformers 4.46.3, Tokenizers 0.20.3, and Flash Attention 2.7.3.

Editorial Comments

DeepSeek OCR is a practical step for document AI, it treats pages as compact optical carriers that reduce decoder sequence length without discarding most information, the model card and technical report describe 97 percent decoding precision at about 10 times compression on Fox benchmark, which is the key claim to test in real workloads. The released model is a 3B MoE decoder with a DeepEncoder front end, packaged for Transformers, with tested versions for PyTorch 2.6.0, CUDA 11.8, and Flash Attention 2.7.3, which lowers setup cost for engineers. The repository shows a single 6.67 GB safetensors shard, which suits common GPUs. Overall, DeepSeek OCR operationalizes optical context compression with a 3B MoE decoder, reports about 97% decoding precision at 10x compression on Fox, provides explicit token budget modes, and includes a tested Transformers setup, validate the throughput claim in your own pipeline.

Check out the Technical Paper, Model on HF and GitHub Repo. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post DeepSeek Just Released a 3B OCR Model: A 3B VLM Designed for High-Performance OCR and Structured Document Conversion appeared first on MarkTechPost.

The landscape of AI is expanding. Today, many of the most powerful LLMs (large language models) reside primarily in the cloud, offering incredible capabilities but also concerns about privacy and limitations around how many files you can upload or how long they stay loaded. Now, a powerful new paradigm is emerging.

This is the dawn of local, private AI.

Imagine a university student preparing for finals with a semester’s overload of data: dozens of  lecture recordings, scanned textbooks, proprietary lab simulations, and folders filled with dozens of handwritten notes. Uploading this massive, copyrighted, and disorganized dataset to the cloud is impractical, and most services would require you to re-upload it for every session. Instead, students are using local LLMs to load all these files and maintain complete control on their laptop.

They prompt the AI: “Analyze my notes on ‘XL1 reactions,’ cross-reference the concept with Professor Dani’s lecture from October 3rd, and explain how it applies to question 5 on the practice exam.”

Seconds later, the AI generates a personalized study guide, highlights the key chemical mechanism from the slides, transcribes the relevant lecture segment, deciphers the student’s handwritten scrawl, and drafts new, targeted practice problems to solidify their understanding.

This switch to local PCs is catalyzed by the release of powerful open models like OpenAI’s new gpt-oss, and supercharged by accelerations provided by NVIDIA RTX AI PCs on LLM frameworks used to run these models locally. A new era of private, instantaneous, and hyper-personalized AI is here.

gpt-oss: the Keys to the Kingdom

OpenAI’s recent launch of gpt-oss is a seismic event for the developer community. It’s a robust 20-billion parameter LLM that is both open-source and, crucially, “open-weight.”

But gpt-oss isn’t just a powerful engine; it’s a meticulously engineered machine with several game-changing features built-in:

● A Specialized Pit Crew (Mixture-of-Experts): The model uses a Mixture-of-Experts (MoE) architecture. Instead of one giant brain doing all the work, it has a team of specialists. For any given task, it intelligently routes the problem to the relevant “experts,” making inference incredibly fast and efficient which is perfect for powering an interactive language-tutor bot, where instant replies are needed to make a practice conversation feel natural and engaging.

● A Tunable Mind (Adjustable Reasoning): The model showcases its thinking with Chain-of-Thought and gives you direct control with adjustable reasoning levels. This allows you to manage the trade-off between speed and depth for any task. For instance, a student writing a term paper could use a “low” setting to quickly summarize a single research article, then switch to “high” to generate a detailed essay outline that thoughtfully synthesizes complex arguments from multiple sources.

● A Marathon Runner’s Memory (Long Context): With a massive 131,000-token context window, it can digest and remember entire technical documents without losing track of the plot. For example, this allows a student to load an entire textbook chapter and all of their lecture notes to prepare for an exam, asking the model to synthesize the key concepts from both sources and generate tailored practice questions.

● Lightweight Power (MXFP4): It is built using MXFP4 quantization. Think of this as building an engine from an advanced, ultra-light alloy. It dramatically reduces the model’s memory footprint, allowing it to deliver high performance. This makes it practical for a computer science student to run a powerful coding assistant directly on their personal laptop in their dorm room, getting help debugging a final project without needing a powerful server or dealing with a slow wifi.

This level of access unlocks superpowers that proprietary cloud models simply can’t match:

● The ‘Air-Gapped’ Advantage (Data Sovereignty): You can analyze and fine-tune LLMs locally using your most sensitive intellectual property without a single byte leaving your secure, air-gapped environment. This is essential for AI data security and compliance (HIPAA/GDPR).

● Forging Specialized AI (Customization): Developers can inject their company’s DNA directly into the model’s brain, teaching it proprietary codebases, specialized industry jargon, or unique creative styles.

● The Zero-Latency Experience (Control): Local deployment provides immediate responsiveness, independent of network connectivity, and offers predictable operational costs.

However, running an engine of this magnitude requires serious computational muscle. To unlock the true potential of gpt-oss, you need hardware built for the job. This model requires at least 16GB of memory to run on local PCs.

The Need for Speed: Why the RTX 50 Series Accelerates Local AI

Benchmarks

When you shift AI processing to your desk, performance isn’t just a metric, it’s the entire experience. It’s the difference between waiting and creating; between a frustrating bottleneck and a seamless thought partner. If you’re waiting for your model to process, you’re losing your creative flow and your analytical edge.

To achieve this seamless experience, the software stack is just as crucial as the hardware. Open-source frameworks like Llama.cpp are essential, acting as the high-performance runtime for these LLMs. Through deep collaboration with NVIDIA, Llama.cpp is heavily optimized for GeForce RTX GPUs for maximum throughput.

The results of this optimization are staggering. Benchmarks utilizing Llama.cpp show NVIDIA’s flagship consumer GPU, the GeForce RTX 5090 , running the gpt-oss-20b model at a blistering 282 tokens per second (tok/s). Tokens are the chunks of text a model processes in a single step, and this metric measures how quickly the AI can generate a response. To put this in perspective, the RTX 5090 significantly outpaces the Mac M3 Ultra (116 tok/s) and AMD’s 7900 XTX (102 tok/s). This performance lead is driven by the dedicated AI hardware, the Tensor Cores, built into the GeForce RTX 5090, specifically engineered to accelerate these demanding AI tasks.

But access isn’t just for developers comfortable with command-line tools. The ecosystem is rapidly evolving to become more user-friendly while leveraging these same NVIDIA optimizations. Applications like LM Studio, which is built on top of Llama.cpp, provide an intuitive interface for running and experimenting with local LLMs. LM Studio makes the process easy and supports advanced techniques like RAG (retrieval-augmented generation).

Ollama is another popular, open-source framework that handles model downloads, environment setup and GPU acceleration automatically,  and multi-model management with seamless application integration. NVIDIA has also collaborated with Ollama to optimize its performance, ensuring these accelerations apply to gpt-oss models. Users can interact directly through the new Ollama app or utilize third-party applications such as AnythingLLM, which offers a streamlined, local interface and also includes support for RAG.

The NVIDIA RTX AI Ecosystem: The Force Multiplier

NVIDIA’s advantage isn’t just about raw power; it’s about the robust, optimized software ecosystem acting as a force multiplier for the hardware, making advanced AI possible on local PCs.

The Democratization of Fine-Tuning: Unsloth AI and RTX

Customizing a 20B model has traditionally required extensive data center resources. However RTX GPUs changed that, and software innovations like Unsloth AI are maximizing this potential.

Optimized for NVIDIA architecture, it leverages techniques like LoRA (Low-Rank Adaptation) to drastically reduce memory usage and increase training speed.

Critically, Unsloth is heavily optimized for the new GeForce RTX 50 Series (Blackwell architecture). This synergy means developers can rapidly fine-tune gpt-oss right on their local PC, fundamentally changing the economics and security of training models on a proprietary “IP vault.”

The Future of AI: Local, Personalized, and Powered by RTX

The release of OpenAI’s gpt-oss is a landmark moment, signaling an industry-wide pivot toward transparency and control. But harnessing this power, achieving instantaneous insights, zero-latency creativity, and ironclad security, requires the right platform.This isn’t just about faster PCs; it’s about a fundamental shift in control and the democratization of AI power. With unmatched performance, and groundbreaking optimization tools like Unsloth AI, NVIDIA RTX AI PCs are essential hardware for this revolution.

Thanks to the NVIDIA AI team for the thought leadership/ Resources for this article. NVIDIA AI team has supported this content/article.
The post The Local AI Revolution: Expanding Generative AI with GPT-OSS-20B and the NVIDIA RTX AI PC appeared first on MarkTechPost.

While a basic Large Language Model (LLM) agent—one that repeatedly calls external tools—is easy to create, these agents often struggle with long and complex tasks because they lack the ability to plan ahead and manage their work over time. They can be considered “shallow” in their execution.

The deepagents library is designed to overcome this limitation by implementing a general architecture inspired by advanced applications like Deep Research and Claude Code.

This architecture gives agents more depth by combining four key features:

A Planning Tool: Allows the agent to strategically break down a complex task into manageable steps before acting.

Sub-Agents: Enables the main agent to delegate specialized parts of the task to smaller, focused agents.

Access to a File System: Provides persistent memory for saving work-in-progress, notes, and final outputs, allowing the agent to continue where it left off.

A Detailed Prompt: Gives the agent clear instructions, context, and constraints for its long-term objectives.

By providing these foundational components, deepagents makes it easier for developers to build powerful, general-purpose agents that can plan, manage state, and execute complex workflows effectively.

In this article, we’ll take a look at a practical example to see how DeepAgents actually work in action. Check out the FULL CODES here.

Core Capabilities of DeepAgents

1. Planning and Task Breakdown: DeepAgents come with a built-in write_todos tool that helps agents break large tasks into smaller, manageable steps. They can track their progress and adjust the plan as they learn new information.

2. Context Management: Using file tools like ls, read_file, write_file, and edit_file, agents can store information outside their short-term memory. This prevents context overflow and lets them handle larger or more detailed tasks smoothly.

3. Sub-Agent Creation: The built-in task tool allows an agent to create smaller, focused sub-agents. These sub-agents work on specific parts of a problem without cluttering the main agent’s context.

4. Long-Term Memory: With support from LangGraph’s Store, agents can remember information across sessions. This means they can recall past work, continue previous conversations, and build on earlier progress.

Setting up dependencies

Copy CodeCopiedUse a different Browser!pip install deepagents tavily-python langchain-google-genai langchain-openai

Environment Variables

In this tutorial, we’ll use the OpenAI API key to power our Deep Agent. However, for reference, we’ll also show how you can use a Gemini model instead.

You’re free to choose any model provider you prefer — OpenAI, Gemini, Anthropic, or others — as DeepAgents works seamlessly with different backends. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserimport os
from getpass import getpass
os.environ[‘TAVILY_API_KEY’] = getpass(‘Enter Tavily API Key: ‘)
os.environ[‘OPENAI_API_KEY’] = getpass(‘Enter OpenAI API Key: ‘)
os.environ[‘GOOGLE_API_KEY’] = getpass(‘Enter Google API Key: ‘)

Importing the necessary libraries

Copy CodeCopiedUse a different Browserimport os
from typing import Literal
from tavily import TavilyClient
from deepagents import create_deep_agent

tavily_client = TavilyClient()

Tools

Just like regular tool-using agents, a Deep Agent can also be equipped with a set of tools to help it perform tasks.

In this example, we’ll give our agent access to a Tavily Search tool, which it can use to gather real-time information from the web. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserfrom typing import Literal
from langchain.chat_models import init_chat_model
from deepagents import create_deep_agent

def internet_search(
query: str,
max_results: int = 5,
topic: Literal[“general”, “news”, “finance”] = “general”,
include_raw_content: bool = False,
):
“””Run a web search”””
search_docs = tavily_client.search(
query,
max_results=max_results,
include_raw_content=include_raw_content,
topic=topic,
)
return search_docs

Sub-Agents

Subagents are one of the most powerful features of Deep Agents. They allow the main agent to delegate specific parts of a complex task to smaller, specialized agents — each with its own focus, tools, and instructions. This helps keep the main agent’s context clean and organized while still allowing for deep, focused work on individual subtasks.

In our example, we defined two subagents:

policy-research-agent — a specialized researcher that conducts in-depth analysis on AI policies, regulations, and ethical frameworks worldwide. It uses the internet_search tool to gather real-time information and produces a well-structured, professional report.

policy-critique-agent — an editorial agent responsible for reviewing the generated report for accuracy, completeness, and tone. It ensures that the research is balanced, factual, and aligned with regional legal frameworks.

Together, these subagents enable the main Deep Agent to perform research, analysis, and quality review in a structured, modular workflow. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browsersub_research_prompt = “””
You are a specialized AI policy researcher.
Conduct in-depth research on government policies, global regulations, and ethical frameworks related to artificial intelligence.

Your answer should:
– Provide key updates and trends
– Include relevant sources and laws (e.g., EU AI Act, U.S. Executive Orders)
– Compare global approaches when relevant
– Be written in clear, professional language

Only your FINAL message will be passed back to the main agent.
“””

research_sub_agent = {
“name”: “policy-research-agent”,
“description”: “Used to research specific AI policy and regulation questions in depth.”,
“system_prompt”: sub_research_prompt,
“tools”: [internet_search],
}

sub_critique_prompt = “””
You are a policy editor reviewing a report on AI governance.
Check the report at `final_report.md` and the question at `question.txt`.

Focus on:
– Accuracy and completeness of legal information
– Proper citation of policy documents
– Balanced analysis of regional differences
– Clarity and neutrality of tone

Provide constructive feedback, but do NOT modify the report directly.
“””

critique_sub_agent = {
“name”: “policy-critique-agent”,
“description”: “Critiques AI policy research reports for completeness, clarity, and accuracy.”,
“system_prompt”: sub_critique_prompt,
}

System Prompt

Deep Agents include a built-in system prompt that serves as their core set of instructions. This prompt is inspired by the system prompt used in Claude Code and is designed to be more general-purpose, providing guidance on how to use built-in tools like planning, file system operations, and subagent coordination.

However, while the default system prompt makes Deep Agents capable out of the box, it’s highly recommended to define a custom system prompt tailored to your specific use case. Prompt design plays a crucial role in shaping the agent’s reasoning, structure, and overall performance.

In our example, we defined a custom prompt called policy_research_instructions, which transforms the agent into an expert AI policy researcher. It clearly outlines a step-by-step workflow — saving the question, using the research subagent for analysis, writing the report, and optionally invoking the critique subagent for review. It also enforces best practices such as Markdown formatting, citation style, and professional tone to ensure the final report meets high-quality policy standards. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserpolicy_research_instructions = “””
You are an expert AI policy researcher and analyst.
Your job is to investigate questions related to global AI regulation, ethics, and governance frameworks.

1️⃣ Save the user’s question to `question.txt`
2️⃣ Use the `policy-research-agent` to perform in-depth research
3️⃣ Write a detailed report to `final_report.md`
4️⃣ Optionally, ask the `policy-critique-agent` to critique your draft
5️⃣ Revise if necessary, then output the final, comprehensive report

When writing the final report:
– Use Markdown with clear sections (## for each)
– Include citations in [Title](URL) format
– Add a ### Sources section at the end
– Write in professional, neutral tone suitable for policy briefings
“””

Main Agent

Here we define our main Deep Agent using the create_deep_agent() function. We initialize the model with OpenAI’s gpt-4o, but as shown in the commented-out line, you can easily switch to Google’s Gemini 2.5 Flash model if you prefer. The agent is configured with the internet_search tool, our custom policy_research_instructions system prompt, and two subagents — one for in-depth research and another for critique.

By default, DeepAgents internally uses Claude Sonnet 4.5 as its model if none is explicitly specified, but the library allows full flexibility to integrate OpenAI, Gemini, Anthropic, or other LLMs supported by LangChain. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browsermodel = init_chat_model(model=”openai:gpt-4o”)
# model = init_chat_model(model=”google_genai:gemini-2.5-flash”)
agent = create_deep_agent(
model=model,
tools=[internet_search],
system_prompt=policy_research_instructions,
subagents=[research_sub_agent, critique_sub_agent],
)

Invoking the Agent

Copy CodeCopiedUse a different Browserquery = “What are the latest updates on the EU AI Act and its global impact?”
result = agent.invoke({“messages”: [{“role”: “user”, “content”: query}]})

Check out the FULL CODES here. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post Meet LangChain’s DeepAgents Library and a Practical Example to See How DeepAgents Actually Work in Action appeared first on MarkTechPost.

Researchers from Stanford, EPFL, and UNC introduce Weak-for-Strong Harnessing, W4S, a new Reinforcement Learning RL framework that trains a small meta-agent to design and refine code workflows that call a stronger executor model. The meta-agent does not fine tune the strong model, it learns to orchestrate it. W4S formalizes workflow design as a multi turn Markov decision process, and trains the meta-agent with a method called Reinforcement Learning for Agentic Workflow Optimization, RLAO. The research team reports consistent gains across 11 benchmarks with a 7B meta-agent trained for about 1 GPU hour.

https://arxiv.org/pdf/2504.04785

W4S operates in turns. The state contains task instructions, the current workflow program, and feedback from prior executions. An action has 2 components, an analysis of what to change, and new Python workflow code that implements those changes. The environment executes the code on validation items, returns accuracy and failure cases, and provides a new state for the next turn. The meta-agent can run a quick self check on one sample, if errors arise it attempts up to 3 repairs, if errors persist the action is skipped. This loop gives learning signal without touching the weights of the strong executor.

https://arxiv.org/pdf/2504.04785

W4S runs as an iterative loop

Workflow generation: The weak meta agent writes a new workflow that leverages the strong model, expressed as executable Python code.

Execution and feedback: The strong model executes the workflow on validation samples, then returns accuracy and error cases as feedback.

Refinement: The meta agent uses the feedback to update the analysis and the workflow, then repeats the loop.

Reinforcement Learning for Agentic Workflow Optimization (RLAO)

RLAO is an offline reinforcement learning procedure over multi turn trajectories. At each iteration, the system samples multiple candidate actions, keeps the best performing action to advance the state, and stores the others for training. The policy is optimized with reward weighted regression. The reward is sparse and compares current validation accuracy to history, a higher weight is given when the new result beats the previous best, a smaller weight is given when it beats the last iteration. This objective favors steady progress while controlling exploration cost.

https://arxiv.org/pdf/2504.04785

Understanding the Results

On HumanEval with GPT-4o-mini as executor, W4S achieves Pass@1 of 95.4, with about 33 minutes of workflow optimization, zero meta-agent API cost, an optimization execution cost of about 0.4 dollars, and about 2.7 minutes to execute the test set at about 0.5 dollars, for a total of about 0.9 dollars. Under the same executor, AFlow and ADAS trail this number. The reported average gains against the strongest automated baseline range from 2.9% to 24.6% across 11 benchmarks.

On math transfer, the meta-agent is trained on GSM Plus and MGSM with GPT-3.5-Turbo as executor, then evaluated on GSM8K, GSM Hard, and SVAMP. The paper reports 86.5 on GSM8K and 61.8 on GSM Hard, both above automated baselines. This indicates that the learned orchestration transfers to related tasks without re training the executor.

Across seen tasks with GPT-4o-mini as executor, W4S surpasses training free automated methods that do not learn a planner. The study also runs ablations where the meta-agent is trained by supervised fine tuning rather than RLAO, the RLAO agent yields better accuracy under the same compute budget. The research team include a GRPO baseline on a 7B weak model for GSM Hard, W4S outperforms it under limited compute.

Iteration budgets matter. The research team sets W4S to about 10 optimization turns on main tables, while AFlow runs about 20 turns and ADAS runs about 30 turns. Despite fewer turns, W4S achieves higher accuracy. This suggests that learned planning over code, combined with validation feedback, makes the search more sample efficient.

https://arxiv.org/pdf/2504.04785

Key Takeaways

W4S trains a 7B weak meta agent with RLAO to write Python workflows that harness stronger executors, modeled as a multi turn MDP.

On HumanEval with GPT 4o mini as executor, W4S reaches Pass@1 of 95.4, with about 33 minutes optimization and about 0.9 dollars total cost, beating automated baselines under the same executor.

Across 11 benchmarks, W4S improves over the strongest baseline by 2.9% to 24.6%, while avoiding fine tuning of the strong model.

The method runs an iterative loop, it generates a workflow, executes it on validation data, then refines it using feedback.

ADAS and AFlow also program or search over code workflows, W4S differs by training a planner with offline reinforcement learning.

Editorial Comments

W4S targets orchestration, not model weights, and trains a 7B meta agent to program workflows that call stronger executors. W4S formalizes workflow design as a multi turn MDP and optimizes the planner with RLAO using offline trajectories and reward weighted regression. Reported results show Pass@1 of 95.4 on HumanEval with GPT 4o mini, average gains of 2.9% to 24.6% across 11 benchmarks, and about 1 GPU hour of training for the meta agent. The framing compares cleanly with ADAS and AFlow, which search agent designs or code graphs, while W4S fixes the executor and learns the planner.

Check out the Technical Paper and GitHub Repo. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post Weak-for-Strong (W4S): A Novel Reinforcement Learning Algorithm that Trains a weak Meta Agent to Design Agentic Workflows with Stronger LLMs appeared first on MarkTechPost.

Microsoft Research proposes BitNet Distillation, a pipeline that converts existing full precision LLMs into 1.58 bit BitNet students for specific tasks, while keeping accuracy close to the FP16 teacher and improving CPU efficiency. The method combines SubLN based architectural refinement, continued pre training, and dual signal distillation from logits and multi head attention relations. Reported results show up to 10× memory savings and about 2.65× faster CPU inference, with task metrics comparable to FP16 across multiple sizes.

What BitNet Distillation changes?

The community already showed that BitNet b1.58 can match full precision quality when trained from scratch, but converting a pretrained FP16 model directly to 1.58 bit often loses accuracy, and the gap grows as model size increases. BitNet Distillation targets this conversion problem for practical downstream deployment. It is designed to preserve accuracy while delivering CPU friendly ternary weights with INT8 activations.

Stage 1: Modeling refinement with SubLN

Low bit models suffer from large activation variance. The research team inserts SubLN normalization inside each Transformer block, specifically before the output projection of the MHSA module and before the output projection of the FFN. This stabilizes hidden state scales that flow into quantized projections, which improves optimization and convergence once weights are ternary. The training loss curves in the analysis section support this design.

Stage 2: Continued pre training to adapt weight distributions

Direct task fine tuning at 1.58 bit gives the student only a small number of task tokens, which is not enough to reshape the FP16 weight distribution for ternary constraints. BitNet Distillation performs a short continued pre training on a general corpus, the research team uses 10B tokens from the FALCON corpus, to push weights toward BitNet like distributions. The visualization shows the mass concentrating near transition boundaries, which makes small gradients flip weights among [-1, 0, 1] during downstream task training. This improves learning capacity without a full pretraining run.

Stage 3: Distillation based fine tuning with two signals

The student learns from the FP16 teacher using logits distillation and multi head self attention relation distillation. The logits path uses temperature softened KL between teacher and student token distributions. The attention path follows the MiniLM and MiniLMv2 formulations, which transfer relations among Q, K, V without requiring the same number of heads, and let you choose a single layer to distill. Ablations show that combining both signals works best, and that selecting one well chosen layer preserves flexibility.

Understanding the results

The research team evaluates classification, MNLI, QNLI, SST 2, and summarization on CNN/DailyMail dataset. It compares three settings, FP16 task fine tuning, direct 1.58 bit task fine tuning, and BitNet Distillation. Figure 1 shows that BitNet Distillation matches FP16 accuracy for Qwen3 backbones at 0.6B, 1.7B, 4B, while the direct 1.58 bit baseline lags more as model size grows. On CPU, tokens per second improve by about 2.65×, and memory drops by about 10× for the student. The research team quantizes activations to INT8 and uses the Straight Through Estimator for gradients through the quantizer.

https://arxiv.org/pdf/2510.13998

The framework is compatible with post training quantization methods such as GPTQ and AWQ, which provide additional gains on top of the pipeline. Distilling from a stronger teacher helps more, which suggests pairing small 1.58 bit students with larger FP16 teachers when available.

Key Takeaways

BitNet Distillation is a 3 stage pipeline, SubLN insertion, continued pre training, and dual distillation from logits and multi head attention relations.

The research reports near FP16 accuracy with about 10× lower memory and about 2.65× faster CPU inference for 1.58 bit students.

The method transfers attention relations using MiniLM and MiniLMv2 style objectives, which do not require matching head counts.

Evaluations cover MNLI, QNLI, SST 2, and CNN/ DailyMail, and include Qwen3 backbones at 0.6B, 1.7B, and 4B parameters.

Deployment targets ternary weights with INT8 activations, with optimized CPU and GPU kernels available in the official BitNet repository.

Editorial Comments

BitNet Distillation is a pragmatic step toward 1.58 bit deployment without a full retrain, the three stage design, SubLN, continual pre training, and MiniLM family attention distillation, maps cleanly to known failure modes in extreme quantization. The reported 10× memory reduction and about 2.65× CPU speedup at near FP16 accuracy indicate solid engineering value for on premise and edge targets. The reliance on attention relation distillation is well grounded in prior MiniLM work, which helps explain the stability of results. The presence of bitnet.cpp with optimized CPU and GPU kernels lowers integration risk for production teams.

Check out the Technical Paper and GitHub Repo. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post Microsoft AI Proposes BitNet Distillation (BitDistill): A Lightweight Pipeline that Delivers up to 10x Memory Savings and about 2.65x CPU Speedup appeared first on MarkTechPost.

Are your LLM code benchmarks actually rejecting wrong-complexity solutions and interactive-protocol violations, or are they passing under-specified unit tests? A team of researchers from UCSD, NYU, University of Washington, Princeton University, Canyon Crest Academy, OpenAI, UC Berkeley, MIT, University of Waterloo, and Sentient Labs introduce AutoCode, a new AI framework that lets LLMs create and verify competitive programming problems, mirroring the workflow of human problem setters. AutoCode reframes evaluation for code-reasoning models by treating problem setting (not only problem solving) as the target task. The system trains LLMs to produce competition-grade statements, test data, and verdict logic that match official online judges at high rates. On a 7,538-problem benchmark built from prior datasets, AutoCode achieves 91.1% consistency with official judgments (FPR 3.7%, FNR 14.1%). On a separate, more difficult 720 recent Codeforces problems (including interactive tasks), the full framework reports 98.7% consistency, 1.3% FPR, 1.2% FNR.

https://arxiv.org/pdf/2510.12803

Why problem setting matters for evaluation?

Public code benchmarks often rely on under-specified tests that let wrong-complexity or shortcut solutions pass. That inflates scores and pollutes reinforcement signals (rewarding fragile tactics). AutoCode’s validator-first approach and adversarial test generation aim to reduce false positives (FPR)—incorrect programs that pass—and false negatives (FNR)—correct programs rejected due to malformed inputs.

https://arxiv.org/pdf/2510.12803

The core loop: Validator → Generator → Checker

AutoCode runs a closed loop that mirrors human contest workflows, but each step is selected from LLM-generated candidates using targeted in-framework tests.

1) Validator (minimize FNR by enforcing input legality)

The system first asks an LLM to synthesize 40 evaluation inputs—10 valid and 30 near-valid illegal (e.g., off-by-one boundary violations). It then prompts the LLM for three candidate validator programs and selects the one that best classifies these cases. This prevents “correct” solutions from crashing on malformed data.

https://arxiv.org/pdf/2510.12803

2) Generator (reduce FPR by adversarial coverage)

Three complementary strategies produce test cases:• Small-data exhaustion for boundary coverage,• Randomized + extreme cases (overflows, precision, hash-collisions),• TLE-inducing structures to break wrong-complexity solutions.

Invalid cases are filtered by the selected validator; then cases are deduplicated and bucket-balanced before sampling.

https://arxiv.org/pdf/2510.12803

3) Checker (verdict logic)

The checker compares contestant outputs with the reference solution under complex rules. AutoCode again generates 40 checker scenarios and three candidate checker programs, keeps only scenarios with validator-approved inputs, and selects the best checker by accuracy against the 40 labeled scenarios.

https://arxiv.org/pdf/2510.12803

4) Interactor (for interactive problems)

For tasks that require dialogue with the judge, AutoCode introduces a mutant-based interactor: it makes small logical edits (“mutants”) to the reference solution, selects interactors that accept the true solution but reject the mutants, maximizing discrimination. This addresses a gap in earlier public datasets that avoided interactives.

https://arxiv.org/pdf/2510.12803

Dual verification enables new problems (not just tests for existing ones)

AutoCode can generate novel problem variants starting from a random “seed” Codeforces problem (<2200 Elo). The LLM drafts a new statement and two solutions: an efficient reference and a simpler brute-force baseline. A problem is accepted only if the reference output matches brute force across the generated test suite (the brute force may TLE on large cases but serves as ground truth on small/exhaustive cases). This dual-verification protocol filters ~27% of error-prone items, lifting reference-solution correctness from 86% → 94% before human review.

Human experts then grade the survivors on solvability, solution correctness, quality, novelty, difficulty. After filtering, 61.6% are usable for model training, 76.3% for human training, and 3.2% are ICPC/IOI-level problems. Difficulty typically increases relative to the seed, and difficulty gain correlates with perceived quality.

https://arxiv.org/pdf/2510.12803

Understanding the results

Existing problems (7,538 total; 195,988 human submissions). AutoCode: 91.1% consistency, 3.7% FPR, 14.1% FNR, vs 72.9–81.0% consistency for prior generators (CodeContests, CodeContests+, TACO, HardTests).

Recent Codeforces problems (720, unfiltered; includes interactives). AutoCode: 98.7% consistency, 1.3% FPR, 1.2% FNR. Ablations show all three generator strategies and prompt optimization contribute: removing prompt optimization drops consistency to 98.0% and more than doubles FNR to 2.9%.

https://arxiv.org/pdf/2510.12803

Key Takeaways

AutoCode couples a Validator–Generator–Checker (+Interactor) loop with dual verification (reference vs. brute-force) to build contest-grade test suites and new problems.

On held-out problems, AutoCode’s test suites reach ~99% consistency with official judges, surpassing prior generators like HardTests (<81%).

For recent Codeforces tasks (including interactives), the full framework reports ~98.7% consistency with ~1.3% FPR and ~1.2% FNR.

The mutant-based interactor reliably accepts the true solution while rejecting mutated variants, improving evaluation for interactive problems.

Human experts rate a sizable fraction of AutoCode-generated items as training-usable and a non-trivial share as contest-quality, aligning with the LiveCodeBench Pro program’s aims.

Editorial Comments

AutoCode is a practical fix for current code benchmarks. It centers problem setting and uses a closed-loop Validator–Generator–Checker (+Interactor) pipeline with dual verification (reference vs. brute-force). This structure reduces false positives/negatives and yields judge-aligned consistency (≈99% on held-out problems; 98.7% on recent Codeforces, including interactives). The approach standardizes constraint legality, adversarial coverage, and protocol-aware judging, which makes downstream RL reward signals cleaner. Its placement under LiveCodeBench Pro fits a hallucination-resistant evaluation program that emphasizes expert-checked rigor.

Check out the Paper and Project. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post AutoCode: A New AI Framework that Lets LLMs Create and Verify Competitive Programming Problems, Mirroring the Workflow of Human Problem Setters appeared first on MarkTechPost.