Author: i-genie

How do you audit frontier LLMs for misaligned behavior in realistic multi-turn, tool-use settings—at scale and beyond coarse aggregate scores? Anthropic released Petri (Parallel Exploration Tool for Risky Interactions), an open-source framework that automates alignment audits by orchestrating an auditor agent to probe a target model across multi-turn, tool-augmented interactions and a judge model to score transcripts on safety-relevant dimensions. In a pilot, Petri was applied to 14 frontier models using 111 seed instructions, eliciting misaligned behaviors including autonomous deception, oversight subversion, whistleblowing, and cooperation with human misuse.

https://alignment.anthropic.com/2025/petri/

What Petri does (at a systems level)?

Petri programmatically: (1) synthesizes realistic environments and tools; (2) drives multi-turn audits with an auditor that can send user messages, set system prompts, create synthetic tools, simulate tool outputs, roll back to explore branches, optionally prefill target responses (API-permitting), and early-terminate; and (3) scores outcomes via an LLM judge across a default 36-dimension rubric with an accompanying transcript viewer.

The stack is built on the UK AI Safety Institute’s Inspect evaluation framework, enabling role binding of auditor, target, and judge in the CLI and support for major model APIs.

https://alignment.anthropic.com/2025/petri/

Pilot results

Anthropic characterizes the release as a broad-coverage pilot, not a definitive benchmark. In the technical report, Claude Sonnet 4.5 and GPT-5 “roughly tie” for strongest safety profile across most dimensions, with both rarely cooperating with misuse; the research overview page summarizes Sonnet 4.5 as slightly ahead on the aggregate “misaligned behavior” score.

A case study on whistleblowing shows models sometimes escalate to external reporting when granted autonomy and broad access—even in scenarios framed as harmless (e.g., dumping clean water)—suggesting sensitivity to narrative cues rather than calibrated harm assessment.

https://alignment.anthropic.com/2025/petri/

Key Takeaways

Scope & behaviors surfaced: Petri was run on 14 frontier models with 111 seed instructions, eliciting autonomous deception, oversight subversion, whistleblowing, and cooperation with human misuse.

System design: An auditor agent probes a target across multi-turn, tool-augmented scenarios (send messages, set system prompts, create/simulate tools, rollback, prefill, early-terminate), while a judge scores transcripts across a default rubric; Petri automates environment setup through to initial analysis.

Results framing: On pilot runs, Claude Sonnet 4.5 and GPT-5 roughly tie for the strongest safety profile across most dimensions; scores are relative signals, not absolute guarantees.

Whistleblowing case study: Models sometimes escalated to external reporting even when the “wrongdoing” was explicitly benign (e.g., dumping clean water), indicating sensitivity to narrative cues and scenario framing.

Stack & limits: Built atop the UK AISI Inspect framework; Petri ships open-source (MIT) with CLI/docs/viewer. Known gaps include no code-execution tooling and potential judge variance—manual review and customized dimensions are recommended.

https://alignment.anthropic.com/2025/petri/

Editorial Comments

Petri is an MIT-licensed, Inspect-based auditing framework that coordinates an auditor–target–judge loop, ships 111 seed instructions, and scores transcripts on 36 dimensions. Anthropic’s pilot spans 14 models; results are preliminary, with Claude Sonnet 4.5 and GPT-5 roughly tied on safety. Known gaps include lack of code-execution tools and judge variance; transcripts remain the primary evidence.

Check out the Technical Paper, GitHub Page and technical blog. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post Anthropic AI Releases Petri: An Open-Source Framework for Automated Auditing by Using AI Agents to Test the Behaviors of Target Models on Diverse Scenarios appeared first on MarkTechPost.

This post was co-written with Cyril Ovely from Vxceed.
Consumer packaged goods (CPG) companies face a critical challenge in emerging economies: how to effectively retain revenue and grow customer loyalty at scale. Although these companies invest 15–20% of their revenue in trade promotions and retailer loyalty programs, the uptake of these programs has historically remained below 30% due to their complexity and the challenge of addressing individual retailer needs.
Vxceed’s Lighthouse platform tackles this challenge with its innovative loyalty module. Trusted by leading global CPG brands across emerging economies in Southeast Asia, Africa, and the Middle East, Lighthouse provides field sales teams with a cutting-edge, AI-driven toolkit. This solution uses generative AI to create personalized sales pitches based on individual retailer data and trends, helping field representatives effectively engage retailers, address common objections, and boost program adoption.
In this post, we show how Vxceed used Amazon Bedrock to develop this AI-powered multi-agent solution that generates personalized sales pitches for field sales teams at scale.
The challenge: Solving a revenue retention problem for brands
Vxceed operates mostly in the emerging economies. The CPG industry is facing challenges such as constant change, high customer expectations, and low barriers to entry. These challenges are more pronounced in the emerging economies. To combat these challenges, CPG companies worldwide invest 15–20% of their revenue annually in trade promotions, often in the format of loyalty programs to retailers.
The uptake of these loyalty programs, however, has traditionally been lower than 30% due to their complexity and the need to address each individual outlet’s needs. To make this challenge more complex, in emerging economies, these loyalty programs are primarily sold through the field sales team, who also act in the role of order capture and fulfilment, and the scale of their operation often spans across millions of outlets. To uplift the loyalty programs uptake, which in turn uplifts the brands revenue retention, the loyalty programs needed to be tailored at a personalized level and pitched properly to each outlet.
Vxceed needed a solution to solve this problem at scale, creating unique, personalized loyalty program selling stories tailored for each individual outlet that the field sales team can use to sell the programs.
This challenge led Vxceed to use Amazon Bedrock, a fully managed service that offers a choice of high-performing foundation models (FMs) from leading AI companies through a single API.
Solution overview
To address the challenges of personalization, scale, and putting the solution in the hands of tens of thousands of field sales teams, Vxceed developed Lighthouse Loyalty Selling Story, an AI-powered solution. The Lighthouse Loyalty Selling Story architecture uses Amazon Bedrock, Amazon API Gateway, Amazon DynamoDB, and AWS Lambda to create a secure, scalable, AI-powered selling story generation system. The solution implements a multi-agent architecture, shown in the following figure, where each component operates within the customer’s private AWS environment, maintaining data security, scalability, and intuitive user interactions. The solution architecture is built around several key components that work together to provide a curated sales enablement experience that is unique for each retailer customer:

Salesperson app – A mobile application is used by field sales teams to access compelling program sales pitches and interact with the system through a chat interface. This serves as the primary touchpoint for sales representatives.
API Gateway and security – The solution uses the following security services:

API Gateway serves as the entry point for application interactions.
Security is enforced using AWS Key Management Service (AWS KMS) for encryption and AWS Secrets Manager for secure credentials management.
Amazon Simple Storage Service (Amazon S3) is used for image storage and management.

Intelligent agents – The solution uses the following Lambda based agents:

Orchestration Agent coordinates the overall flow and interaction between components.
Story Framework Agent establishes the narrative structure.
Story Generator Agent creates personalized content.
Story Review Agent maintains quality and compliance with brand guidelines.
Brand Guidelines Agent maintains brand consistency.
Business Rules Agent enforces business logic and constraints.

Data services layer – The data services layer consists of the following components:

Data API services provide access to critical business information, including:

Outlet profile data
Loyalty program details
Historical data
Purchase profile information

Integration with Lighthouse artificial intelligence and machine learning (AI/ML) models and data lake for advanced analytics.
Amazon Bedrock Knowledge Bases for enhanced context and information.

Advanced capabilities – The solution offers the following additional capabilities:

Q&A Service enables natural language interactions for sales queries.
CTA (Call-to-Action) Service streamlines the retail outlet signup process.
An Amazon Bedrock large language model (LLM) powers intelligent responses.
Amazon Bedrock Guardrails facilitates appropriate and compliance-aligned interactions.

The architecture implements a secure, scalable, and serverless design that uses AWS managed services to deliver a sophisticated sales enablement solution.
Multi-agent AI architecture for secure orchestration
Vxceed built a multi-agent AI system on Lambda to manage personalized sales storytelling. The architecture comprises specialized agents that work together to create, validate, and deliver compelling sales pitches while maintaining alignment with business rules and brand guidelines.
The following is a detailed breakdown of the multi-agent AI architecture:

Orchestration Agent – Coordinates the workflow between agents and manages the overall story creation process, interfacing with the Amazon Bedrock LLM for intelligent processing.
Story Framework Agent – Establishes the narrative structure and flow of sales pitches based on proven storytelling patterns and sales methodologies.
Story Generator Agent – Creates personalized content by combining data from multiple sources, including outlet profiles, loyalty program details, and historical data.
Story Review Agent – Validates generated content for accuracy, completeness, and effectiveness before delivery to sales personnel.
Brand Guidelines Agent – Makes sure generated content adheres to brand voice, tone, and visual standards.
Business Rules Agent – Enforces business logic, customer brand compliance requirements, and operational constraints across generated content.

Each agent is implemented as a serverless Lambda function, enabling scalable and cost-effective processing while maintaining strict security controls through integration with AWS KMS and Secrets Manager. The agents interact with the Amazon Bedrock LLM and guardrails to provide appropriate and responsible AI-generated content.
Guardrails
Lighthouse uses Amazon Bedrock Guardrails to maintain professional, focused interactions. The system uses denied topics and word filters to help prevent unrelated discussions and unprofessional language, making sure conversations remain centered on customer needs. These guardrails screen out inappropriate content, establish clear boundaries around sensitive topics, and diplomatically address competitive inquiries while staying aligned with organizational values.
Why Vxceed chose Amazon Bedrock
Vxceed selected Amazon Bedrock over other AI solutions because of four key advantages:

Enterprise-grade security and privacy – With Amazon Bedrock, you can configure your AI workloads and data so your information remains securely within your own virtual private cloud (VPC). This approach maintains a private, encrypted environment for AI operations, helping keep data protected and isolated within the your VPC. For more details, refer to Security in Amazon Bedrock.
Managed services on AWS – Lighthouse Loyalty Selling Story runs on Vxceed’s existing AWS infrastructure, minimizing integration effort and providing end-to-end control over data and operations using managed services such as Amazon Bedrock.
Access to multiple AI models – Amazon Bedrock supports various FMs, so Vxceed can experiment and optimize performance across different use cases. Vxceed uses Anthropic’s Claude 3.5 Sonnet for its ability to handle sophisticated conversational interactions and complex language processing tasks.
Robust AI development tools – Vxceed accelerated development by using Amazon Bedrock Knowledge Bases, prompt engineering libraries, and agent frameworks for efficient AI orchestration.

Business impact and future outlook
The implementation delivered significant measurable improvements across three key areas.
Enhanced customer service
The solution achieved a 95% response accuracy rate while automating 90% of loyalty program-related queries. This automation facilitates consistent, accurate responses to customer objections and queries, helping salespeople and significantly improving the retailer experience.
Accelerated revenue growth
Early customer feedback and industry analysis indicate program enrollment increased by 5–15%. This growth demonstrates how removing friction from the enrollment process directly impacts business outcomes.
Improved operational efficiency
The solution delivered substantial operational benefits:

20% reduction in enrolment processing time
10% decrease in support time requirements
Annual savings of 2 person-months per geographical region in administrative overhead

These efficiency gains help Vxceed customers focus on higher-value activities while reducing operational costs. The combination of faster processing and reduced support requirements creates a scalable foundation for program growth.
Conclusion
AWS partnered with Vxceed to support their AI strategy, resulting in the development of Lighthouse Loyalty Selling Story, an innovative personalized sales pitch solution. Using AWS services including Amazon Bedrock and Lambda, Vxceed successfully built a secure, AI-powered solution that creates personalized selling stories at scale for CPG industry field sales teams. Looking ahead, Vxceed plans to further refine Lighthouse Loyalty Selling Story by:

Optimizing AI inference costs to improve scalability and cost-effectiveness
Adding a Language Agent to present the generated selling story in the native language of choice
Adding RAG and GraphRAG to further enhance the story generation effectiveness

With this collaboration, Vxceed aims to significantly improve CPG industry field sales management, delivering secure, efficient, and AI-powered solutions for CPG companies and brands.
If you are interested in implementing a similar AI-powered solution, start by understanding how to implement asynchronous AI agents using Amazon Bedrock. See Creating asynchronous AI agents with Amazon Bedrock to learn about the implementation patterns for multi-agent systems and develop secure, AI-powered solutions for your organization.
About the Authors

Roger Wang is a Senior Solution Architect at AWS. He is a seasoned architect with over 20 years of experience in the software industry. He helps New Zealand and global software and SaaS companies use cutting-edge technology at AWS to solve complex business challenges. Roger is passionate about bridging the gap between business drivers and technological capabilities, and thrives on facilitating conversations that drive impactful results.

Deepika Kumar is a Solutions Architect at AWS. She has over 13 years of experience in the technology industry and has helped enterprises and SaaS organizations build and securely deploy their workloads on the cloud. She is passionate about using generative AI in a responsible manner, whether that is driving product innovation, boosting productivity, or enhancing customer experiences.

Jhalak Modi is a Solution Architect at AWS, specializing in cloud architecture, security, and AI-driven solutions. She helps businesses use AWS to build secure, scalable, and innovative solutions. Passionate about emerging technologies, Jhalak actively shares her expertise in cloud computing, automation, and responsible AI adoption, empowering organizations to accelerate digital transformation and stay ahead in a rapidly evolving tech landscape.

Cyril Ovely, CTO and co-founder of Vxceed Software Solutions, leads the company’s SaaS-based logistics solutions for CPG brands. With 33 years of experience, including 22 years at Vxceed, he previously worked in analytical and process control instrumentation. An engineer by training, Cyril architects Vxceed’s SaaS offerings and drives innovation from his base in Auckland, New Zealand.

Machine learning operations (MLOps) is the combination of people, processes, and technology to productionize ML use cases efficiently. To achieve this, enterprise customers must develop MLOps platforms to support reproducibility, robustness, and end-to-end observability of the ML use case’s lifecycle. Those platforms are based on a multi-account setup by adopting strict security constraints, development best practices such as automatic deployment using continuous integration and delivery (CI/CD) technologies, and permitting users to interact only by committing changes to code repositories. For more information about MLOps best practices, refer to the MLOps foundation roadmap for enterprises with Amazon SageMaker.
Terraform by HashiCorp has been embraced by many customers as the main infrastructure as code (IaC) approach to develop, build, deploy, and standardize AWS infrastructure for multi-cloud solutions. Furthermore, development repositories and CI/CD technologies such as GitHub and GitHub Actions, respectively, have been adopted widely by the DevOps and MLOps community across the world.
In this post, we show how to implement an MLOps platform based on Terraform using GitHub and GitHub Actions for the automatic deployment of ML use cases. Specifically, we deep dive on the necessary infrastructure and show you how to utilize custom Amazon SageMaker Projects templates, which contain example repositories that help data scientists and ML engineers deploy ML services (such as an Amazon SageMaker endpoint or batch transform job) using Terraform. You can find the source code in the following GitHub repository.
Solution overview
The MLOps architecture solution creates the necessary resources to build a comprehensive training pipeline, registering the models in the Amazon SageMaker Model Registry, and its deployment to preproduction and production environments. This foundational infrastructure enables a systematic approach to ML operations, providing a robust framework that streamlines the journey from model development to deployment.
The end-users (data scientists or ML engineers) will select the organization SageMaker Project template that fits their use case. SageMaker Projects helps organizations set up and standardize developer environments for data scientists and CI/CD systems for MLOps engineers. The project deployment creates, from the GitHub templates, a GitHub private repository and CI/CD resources that data scientists can customize according to their use case. Depending on the chosen SageMaker project, other project-specific resources will also be created.

Custom SageMaker Project template
SageMaker projects deploys the associated AWS CloudFormation template of the AWS Service Catalog product to provision and manage the infrastructure and resources required for your project, including the integration with a source code repository.
At the time of writing, four custom SageMaker Projects templates are available for this solution:

MLOps template for LLM training and evaluation – An MLOps pattern that shows a simple one-account Amazon SageMaker Pipelines setup for large language models (LLMs) This template supports fine-tuning and evaluation.
MLOps template for model building and training – An MLOps pattern that shows a simple one-account SageMaker Pipelines setup. This template supports model training and evaluation.
MLOps template for model building, training, and deployment – An MLOps pattern to train models using SageMaker Pipelines and deploy the trained model into preproduction and production accounts. This template supports real-time inference, batch inference pipelines, and bring-your-own-containers (BYOC).
MLOps template for promoting the full ML pipeline across environments – An MLOps pattern to show how to take the same SageMaker pipeline across environments from dev to prod. This template supports a pipeline for batch inference.

Each SageMaker project template has associated GitHub repository templates that are cloned to be used for your use case:

MLOps template for LLM training and evaluation – Associated with the LLM training repository.
MLOps template for model building and training – Associated with the model training repository.
MLOps template for model building, training, and deployment – Associated with the BYOC repository (optional), model training repository, and real time inference repository or batch inference repository.
MLOps template for promoting the full ML pipeline across environments – Associated with pipeline promotion repository.

When a custom SageMaker project is deployed by a data scientist, the associated GitHub template repositories are cloned through an invocation of the AWS Lambda function <prefix>_clone_repo_lambda, which creates a new GitHub repository for your project.

Infrastructure Terraform modules
The Terraform code, found under base-infrastructure/terraform, is structured with reusable modules that are used across different deployment environments. Their instantiation will be found for each environment under base-infrastructure/terraform/<ENV>/main.tf. There are seven key reusable modules:

KMS – Creates an AWS Key Management Service (AWS KMS) key
Lambda – Creates a Lambda function and Amazon CloudWatch log group
Networking – Creates a virtual private cloud (VPC), various subnets, security group, NAT gateway, internet gateway, route table and routes, and multiple VPC endpoints for the networking setup for Amazon SageMaker Studio
S3 – Creates an Amazon Simple Storage Service (Amazon S3) bucket
SageMaker – Creates SageMaker Studio and SageMaker users
SageMaker Roles – Creates AWS Identity and Access Management (IAM) roles for SageMaker Studio
Service Catalog – Creates Service Catalog products from a CloudFormation template

There are also some environment-specific resources, which can be found directly under base-infrastructure/terraform/<ENV>.

Prerequisites
Before you start the deployment process, complete the following three steps:

Prepare AWS accounts to deploy the platform. We recommend using three AWS accounts for three typical MLOps environments: experimentation, preproduction, and production. However, you can deploy the infrastructure to just one account for testing purposes.
Create a GitHub organization.
Create a personal access token (PAT). It is recommended to create a service or platform account and use its PAT.

Bootstrap your AWS accounts for GitHub and Terraform
Before we can deploy the infrastructure, the AWS accounts you have vended need to be bootstrapped. This is required so that Terraform can manage the state of the resources deployed. Terraform backends enable secure, collaborative, and scalable infrastructure management by streamlining version control, locking, and centralized state storage. Therefore, we deploy an S3 bucket and Amazon DynamoDB table for storing states and locking consistency checking.
Bootstrapping is also required so that GitHub can assume a deployment role in your account, therefore we deploy an IAM role and OpenID Connect (OIDC) identity provider (IdP). As an alternative to employing long-lived IAM user access keys, organizations can implement an OIDC IdP within your AWS account. This configuration facilitates the utilization of IAM roles and short-term credentials, enhancing security and adherence to best practices.
You can choose from two options to bootstrap your account: a bootstrap.sh Bash script and a bootstrap.yaml CloudFormation template, both stored at the root of the repository.
Bootstrap using a CloudFormation template
Complete the following steps to use the CloudFormation template:

Make sure the AWS Command Line Interface (AWS CLI) is installed and credentials are loaded for the target account that you want to bootstrap.
Identify the following:

Environment type of the account: dev, preprod, or prod.
Name of your GitHub organization.
(Optional) Customize the S3 bucket name for Terraform state files by choosing a prefix.
(Optional) Customize the DynamoDB table name for state locking.

Run the following command, updating the details from Step 2:

# Update
export ENV=xxx
export GITHUB_ORG=xxx
# Optional
export TerraformStateBucketPrefix=terraform-state
export TerraformStateLockTableName=terraform-state-locks

aws cloudformation create-stack
–stack-name YourStackName
–template-body file://bootstrap.yaml
–capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM
–parameters ParameterKey=Environment,ParameterValue=$ENV
ParameterKey=GitHubOrg,ParameterValue=$GITHUB_ORG
ParameterKey=OIDCProviderArn,ParameterValue=””
ParameterKey=TerraformStateBucketPrefix,ParameterValue=$TerraformStateBucketPrefix
ParameterKey=TerraformStateLockTableName,ParameterValue=$TerraformStateLockTableName

Bootstrap using a Bash script
Complete the following steps to use the Bash script:

Make sure the AWS CLI is installed and credentials are loaded for the target account that you want to bootstrap.
Identify the following:

Environment type of the account: dev, preprod, or prod.
Name of your GitHub organization.
(Optional) Customize the S3 bucket name for Terraform state files by choosing a prefix.
(Optional) Customize the DynamoDB table name for state locking.

Run the script (bash ./bootstrap.sh) and input the details from Step 2 when prompted. You can leave most of these options as default.

If you change the TerraformStateBucketPrefix or TerraformStateLockTableName parameters, you must update the environment variables (S3_PREFIX and DYNAMODB_PREFIX) in the deploy.yml file to match.
Set up your GitHub organization
In the final step before infrastructure deployment, you must configure your GitHub organization by cloning code from this example into specific locations.
Base infrastructure
Create a new repository in your organization that will contain the base infrastructure Terraform code. Give your repository a unique name, and move the code from this example’s base-infrastructure folder into your newly created repository. Make sure the .github folder is also moved to the new repository, which stores the GitHub Actions workflow definitions. GitHub Actions make it possible to automate, customize, and execute your software development workflows right in your repository. In this example, we use GitHub Actions as our preferred CI/CD tooling.
Next, set up some GitHub secrets in your repository. Secrets are variables that you create in an organization, repository, or repository environment. The secrets that you create are available to use in our GitHub Actions workflows. Complete the following steps to create your secrets:

Navigation to the base infrastructure repository.
Choose Settings, Secrets and Variables, and Actions.
Create two secrets:

AWS_ASSUME_ROLE_NAME – This is created in the bootstrap script with the default name aws-github-oidc-role, and should be updated in the secret with whichever role name you choose.
PAT_GITHUB – This is your GitHub PAT token, created in the prerequisite steps.

Template repositories
The template-repos folder of our example contains multiple folders with the seed code for our SageMaker Projects templates. Each folder should be added to your GitHub organization as a private template repository. Complete the following steps:

Create the repository with the same name as the example folder, for every folder in the template-repos directory.
Choose Settings in each newly created repository.
Select the Private Template option.

Make sure you move all the code from the example folder to your private template, including the .github folder.
Update the configuration file
At the root of the base infrastructure folder is a config.json file. This file enables the multi-account, multi-environment mechanism. The example JSON structure is as follows:

{
“environment_name”: {
“region”: “X”,
“dev_account_number”: “XXXXXXXXXXXX”,
“preprod_account_number”: “XXXXXXXXXXXX”,
“prod_account_number”: “XXXXXXXXXXXX”
}
}

For your MLOps environment, simply change the name of environment_name to your desired name, and update the AWS Region and account numbers accordingly. Note the account numbers will correspond to the AWS accounts you bootstrapped. This config.json permits you to vend as many MLOps platforms as you desire. To do so, simply create a new JSON object in the file with the respective environment name, Region, and bootstrapped account numbers. Then locate the GitHub Actions deployment workflow under .github/workflows/deploy.yaml and add your new environment name inside each list object in the matrix key. When we deploy our infrastructure using GitHub Actions, we use a matrix deployment to deploy to all our environments in parallel.
Deploy the infrastructure
Now that you have set up your GitHub organization, you’re ready to deploy the infrastructure into the AWS accounts. Changes to the infrastructure will deploy automatically when changes are made to the main branch, therefore when you make changes to the config file, this should trigger the infrastructure deployment. To launch your first deployment manually, complete the following steps:

Navigate to your base infrastructure repository.
Choose the Actions tab.
Choose Deploy Infrastructure.
Choose Run Workflow and choose your desired branch for deployment.

This will launch the GitHub Actions workflow for deploying the experimentation, preproduction, and production infrastructure in parallel. You can visualize these deployments on the Actions tab.
Now your AWS accounts will contain the necessary infrastructure for your MLOps platform.
End-user experience
The following demonstration illustrates the end-user experience.

Clean up
To delete the multi-account infrastructure created by this example and avoid further charges, complete the following steps:

In the development AWS account, manually delete the SageMaker projects, SageMaker domain, SageMaker user profiles, Amazon Elastic File Service (Amazon EFS) storage, and AWS security groups created by SageMaker.
In the development AWS account, you might need to provide additional permissions to the launch_constraint_role IAM role. This IAM role is used as a launch constraint. Service Catalog will use this permission to delete the provisioned products.
In the development AWS account, manually delete the resources like repositories (Git), pipelines, experiments, model groups, and endpoints created by SageMaker Projects.
For preproduction and production AWS accounts, manually delete the S3 bucket ml-artifacts-<region>-<account-id> and the model deployed through the pipeline.
After you complete these changes, trigger the GitHub workflow for destroying.
If the resources aren’t deleted, manually delete the pending resources.
Delete the IAM user that you created for GitHub Actions.
Delete the secret in AWS Secrets Manager that stores the GitHub personal access token.

Conclusion
In this post, we walked through the process of deploying an MLOps platform based on Terraform and using GitHub and GitHub Actions for the automatic deployment of ML use cases. This solution effectively integrates four custom SageMaker Projects templates for model building, training, evaluation and deployment with specific SageMaker pipelines. In our scenario, we focused on deploying a multi-account and multi-environment MLOps platform. For a comprehensive understanding of the implementation details, visit the GitHub repository.

About the authors
Jordan Grubb is a DevOps Architect at AWS, specializing in MLOps. He enables AWS customers to achieve their business outcomes by delivering automated, scalable, and secure cloud architectures. Jordan is also an inventor, with two patents within software engineering. Outside of work, he enjoys playing most sports, traveling, and has a passion for health and wellness.
Irene Arroyo Delgado is an AI/ML and GenAI Specialist Solution at AWS. She focuses on bringing out the potential of generative AI for each use case and productionizing ML workloads, to achieve customers’ desired business outcomes by automating end-to-end ML lifecycles. In her free time, Irene enjoys traveling and hiking.

Human handoff is a key component of customer service automation—it ensures that when AI reaches its limits, a skilled human can seamlessly take over. In this tutorial, we’ll implement a human handoff system for an AI-powered insurance agent using Parlant. You’ll learn how to create a Streamlit-based interface that allows a human operator (Tier 2) to view live customer messages and respond directly within the same session, bridging the gap between automation and human expertise. Check out the FULL CODES here.

Setting up the dependencies

Make sure you have a valid OpenAI API key before starting. Once you’ve generated it from your OpenAI dashboard, create a .env file in your project’s root directory and store the key securely there like this:

Copy CodeCopiedUse a different BrowserOPENAI_API_KEY=your_api_key_here

This keeps your credentials safe and prevents them from being hardcoded into your codebase.

Copy CodeCopiedUse a different Browserpip install parlant dotenv streamlit

Insurance Agent (agent.py) 

We’ll start by building the agent script, which defines the AI’s behavior, conversation journeys, glossary, and the human handoff mechanism. This will form the core logic that powers our insurance assistant in Parlant. Once the agent is ready and capable of escalating to manual mode, we’ll move on to developing the Streamlit-based human handoff interface, where human operators can view ongoing sessions, read customer messages, and respond in real time — creating a seamless collaboration between AI automation and human expertise. Check out the FULL CODES here.

Loading the required libraries

Copy CodeCopiedUse a different Browserimport asyncio
import os
from datetime import datetime
from dotenv import load_dotenv
import parlant.sdk as p

load_dotenv()

Defining the Agent’s Tools

Copy CodeCopiedUse a different Browser@p.tool
async def get_open_claims(context: p.ToolContext) -> p.ToolResult:
return p.ToolResult(data=[“Claim #123 – Pending”, “Claim #456 – Approved”])

@p.tool
async def file_claim(context: p.ToolContext, claim_details: str) -> p.ToolResult:
return p.ToolResult(data=f”New claim filed: {claim_details}”)

@p.tool
async def get_policy_details(context: p.ToolContext) -> p.ToolResult:
return p.ToolResult(data={
“policy_number”: “POL-7788”,
“coverage”: “Covers accidental damage and theft up to $50,000”
})

The code block introduces three tools that simulate interactions an insurance assistant might need. 

The get_open_claims tool represents an asynchronous function that retrieves a list of open insurance claims, allowing the agent to provide users with up-to-date information about pending or approved claims. 

The file_claim tool accepts claim details as input and simulates the process of filing a new insurance claim, returning a confirmation message to the user. 

Finally, the get_policy_details tool provides essential policy information, such as the policy number and coverage limits, enabling the agent to respond accurately to questions about insurance coverage. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browser@p.tool
async def initiate_human_handoff(context: p.ToolContext, reason: str) -> p.ToolResult:
“””
Initiate handoff to a human agent when the AI cannot adequately help the customer.
“””
print(f” Initiating human handoff: {reason}”)
# Setting session to manual mode stops automatic AI responses
return p.ToolResult(
data=f”Human handoff initiated because: {reason}”,
control={
“mode”: “manual” # Switch session to manual mode
}
)

The initiate_human_handoff tool enables the AI agent to gracefully transfer a conversation to a human operator when it detects that the issue requires human intervention. By switching the session to manual mode, it pauses all automated responses, ensuring the human agent can take full control. This tool helps maintain a smooth transition between AI and human assistance, ensuring complex or sensitive customer queries are handled with the appropriate level of expertise.

Defining the Glossary

A glossary defines key terms and phrases that the AI agent should recognize and respond to consistently. It helps maintain accuracy and brand alignment by giving the agent clear, predefined answers for common domain-specific queries. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserasync def add_domain_glossary(agent: p.Agent):
await agent.create_term(
name=”Customer Service Number”,
description=”You can reach us at +1-555-INSURE”,
)
await agent.create_term(
name=”Operating Hours”,
description=”We are available Mon-Fri, 9AM-6PM”,
)

Defining the Journeys

Copy CodeCopiedUse a different Browser# —————————
# Claim Journey
# —————————

async def create_claim_journey(agent: p.Agent) -> p.Journey:
journey = await agent.create_journey(
title=”File an Insurance Claim”,
description=”Helps customers report and submit a new claim.”,
conditions=[“The customer wants to file a claim”],
)

s0 = await journey.initial_state.transition_to(chat_state=”Ask for accident details”)
s1 = await s0.target.transition_to(tool_state=file_claim, condition=”Customer provides details”)
s2 = await s1.target.transition_to(chat_state=”Confirm claim was submitted”, condition=”Claim successfully created”)
await s2.target.transition_to(state=p.END_JOURNEY, condition=”Customer confirms submission”)

return journey

# —————————
# Policy Journey
# —————————

async def create_policy_journey(agent: p.Agent) -> p.Journey:
journey = await agent.create_journey(
title=”Explain Policy Coverage”,
description=”Retrieves and explains customer’s insurance coverage.”,
conditions=[“The customer asks about their policy”],
)

s0 = await journey.initial_state.transition_to(tool_state=get_policy_details)
await s0.target.transition_to(
chat_state=”Explain the policy coverage clearly”,
condition=”Policy info is available”,
)

await agent.create_guideline(
condition=”Customer presses for legal interpretation of coverage”,
action=”Politely explain that legal advice cannot be provided”,
)
return journey

The Claim Journey guides customers through the process of filing a new insurance claim. It collects accident details, triggers the claim filing tool, confirms successful submission, and then ends the journey—automating the entire claim initiation flow.

The Policy Journey helps customers understand their insurance coverage by retrieving policy details and explaining them clearly. It also includes a guideline to ensure the AI avoids giving legal interpretations, maintaining compliance and professionalism. Check out the FULL CODES here.

Defining the Main Runner

Copy CodeCopiedUse a different Browserasync def main():
async with p.Server() as server:
agent = await server.create_agent(
name=”Insurance Support Agent”,
description=(
“Friendly Tier-1 AI assistant that helps with claims and policy questions. ”
“Escalates complex or unresolved issues to human agents (Tier-2).”
),
)

# Add shared terms & definitions
await add_domain_glossary(agent)

# Journeys
claim_journey = await create_claim_journey(agent)
policy_journey = await create_policy_journey(agent)

# Disambiguation rule
status_obs = await agent.create_observation(
“Customer mentions an issue but doesn’t specify if it’s a claim or policy”
)
await status_obs.disambiguate([claim_journey, policy_journey])

# Global Guidelines
await agent.create_guideline(
condition=”Customer asks about unrelated topics”,
action=”Kindly redirect them to insurance-related support only”,
)

# Human Handoff Guideline
await agent.create_guideline(
condition=”Customer requests human assistance or AI is uncertain about the next step”,
action=”Initiate human handoff and notify Tier-2 support.”,
tools=[initiate_human_handoff],
)

print(” Insurance Support Agent with Human Handoff is ready! Open the Parlant UI to chat.”)

if __name__ == “__main__”:
asyncio.run(main())

Running the Agent

Copy CodeCopiedUse a different Browserpython agent.py

This will start the Parlant agent locally on http://localhost:8800 , where it will handle all conversation logic and session management.

In the next step, we’ll connect this running agent to our Streamlit-based Human Handoff interface, allowing a human operator to seamlessly join and manage live conversations using the Parlant session ID. Check out the FULL CODES here.

Human Handoff (handoff.py) 

Importing Libraries

Copy CodeCopiedUse a different Browserimport asyncio
import streamlit as st
from datetime import datetime
from parlant.client import AsyncParlantClient

Setting Up the Parlant Client

Once the AI agent script is running, Parlant will host its server locally (usually at http://localhost:8800).

Here, we connect to that running instance by creating an asynchronous client. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserclient = AsyncParlantClient(base_url=”http://localhost:8800″)

When you run the agent and get a session ID, we’ll use that ID in this UI to connect and manage that specific conversation.

Session State Management

Streamlit’s session_state is used to persist data across user interactions — such as storing received messages and tracking the latest event offset to fetch new ones efficiently. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserif “events” not in st.session_state:
st.session_state.events = []
if “last_offset” not in st.session_state:
st.session_state.last_offset = 0

Message Rendering Function

This function controls how messages appear in the Streamlit interface — differentiating between customers, AI, and human agents for clarity. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserdef render_message(message, source, participant_name, timestamp):
if source == “customer”:
st.markdown(f”** Customer [{timestamp}]:** {message}”)
elif source == “ai_agent”:
st.markdown(f”** AI [{timestamp}]:** {message}”)
elif source == “human_agent”:
st.markdown(f”** {participant_name} [{timestamp}]:** {message}”)
elif source == “human_agent_on_behalf_of_ai_agent”:
st.markdown(f”** (Human as AI) [{timestamp}]:** {message}”)

Fetching Events from Parlant

This asynchronous function retrieves new messages (events) from Parlant for the given session.

Each event represents a message in the conversation — whether sent by the customer, AI, or human operator. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserasync def fetch_events(session_id):
try:
events = await client.sessions.list_events(
session_id=session_id,
kinds=”message”,
min_offset=st.session_state.last_offset,
wait_for_data=5
)
for event in events:
message = event.data.get(“message”)
source = event.source
participant_name = event.data.get(“participant”, {}).get(“display_name”, “Unknown”)
timestamp = getattr(event, “created”, None) or event.data.get(“created”, “Unknown Time”)
event_id = getattr(event, “id”, “Unknown ID”)

st.session_state.events.append(
(message, source, participant_name, timestamp, event_id)
)
st.session_state.last_offset = max(st.session_state.last_offset, event.offset + 1)

except Exception as e:
st.error(f”Error fetching events: {e}”)

Sending Messages as Human or AI

Two helper functions are defined to send messages:

One as a human operator (source=”human_agent”)

Another as if sent by the AI, but manually triggered by a human (source=”human_agent_on_behalf_of_ai_agent”)

Check out the FULL CODES here.

Copy CodeCopiedUse a different Browser
async def send_human_message(session_id: str, message: str, operator_name: str = “Tier-2 Operator”):
event = await client.sessions.create_event(
session_id=session_id,
kind=”message”,
source=”human_agent”,
message=message,
participant={
“id”: “operator-001”,
“display_name”: operator_name
}
)
return event

async def send_message_as_ai(session_id: str, message: str):
event = await client.sessions.create_event(
session_id=session_id,
kind=”message”,
source=”human_agent_on_behalf_of_ai_agent”,
message=message
)
return event

Streamlit Interface

Finally, we build a simple, interactive Streamlit UI:

Enter a session ID (from the Parlant UI)

View chat history

Send messages as either Human or AI

Refresh to pull new messages

Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserst.title(” Human Handoff Assistant”)

session_id = st.text_input(“Enter Parlant Session ID:”)

if session_id:
st.subheader(“Chat History”)
if st.button(“Refresh Messages”):
asyncio.run(fetch_events(session_id))

for msg, source, participant_name, timestamp, event_id in st.session_state.events:
render_message(msg, source, participant_name, timestamp)

st.subheader(“Send a Message”)
operator_msg = st.text_input(“Type your message:”)

if st.button(“Send as Human”):
if operator_msg.strip():
asyncio.run(send_human_message(session_id, operator_msg))
st.success(“Message sent as human agent “)
asyncio.run(fetch_events(session_id))

if st.button(“Send as AI”):
if operator_msg.strip():
asyncio.run(send_message_as_ai(session_id, operator_msg))
st.success(“Message sent as AI “)
asyncio.run(fetch_events(session_id))

Check out the FULL CODES here. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post Building a Human Handoff Interface for AI-Powered Insurance Agent Using Parlant and Streamlit appeared first on MarkTechPost.

Amazon Bedrock is a fully managed service that offers a choice of high-performing foundation models (FMs) from leading AI companies through a single API, along with capabilities to build generative AI applications with security, privacy, and responsible AI.
Batch inference in Amazon Bedrock is for larger workloads where immediate responses aren’t critical. With a batch processing approach, organizations can analyze substantial datasets efficiently, with significant cost advantages: you can benefit from a 50% reduction in pricing compared to the on-demand option. This makes batch inference particularly valuable for handling extensive data to get inference from Amazon Bedrock FMs.
As organizations scale their use of Amazon Bedrock FMs for large-volume data processing, implementing effective monitoring and management practices for batch inference jobs becomes an important focus area for optimization. This solution demonstrates how to implement automated monitoring for Amazon Bedrock batch inference jobs using AWS serverless services such as AWS Lambda, Amazon DynamoDB, and Amazon EventBridge, reducing operational overhead while maintaining reliable processing of large-scale batch inference workloads. Through a practical example in the financial services sector, we show how to build a production-ready system that automatically tracks job status, provides real-time notifications, and maintains audit records of processing activities.
Solution overview
Consider a scenario where a financial services company manages millions of customer interactions and data points, including credit histories, spending patterns, and financial preferences. This company recognized the potential of using advanced AI capabilities to deliver personalized product recommendations at scale. However, processing such massive datasets in real time isn’t always necessary or cost-effective.
The solution presented in this post uses batch inference in Amazon Bedrock with automated monitoring to process large volumes of customer data efficiently using the following architecture.

This architecture workflow includes the following steps:

The financial services company uploads customer credit data and product data to be processed to an Amazon Simple Storage Service (Amazon S3) bucket.
The first Lambda function reads the prompt template and data from the S3 bucket, and creates a JSONL file with prompts for the customers along with their credit data and available financial products.
The same Lambda function triggers a new Amazon Bedrock batch inference job using this JSONL file.
In the prompt template, the FM is given a role of expert in recommendation systems within the financial services industry. This way, the model understands the customer and their credit information to intelligently recommend most suitable products.
An EventBridge rule monitors the state changes of the batch inference job. When the job completes or fails, the rule triggers a second Lambda function.
The second Lambda function creates an entry for the job with its status in a DynamoDB table.
After a batch job is complete, its output files (containing personalized product recommendations) will be available in the S3 bucket’s inference_results folder.

This automated monitoring solution for Amazon Bedrock batch inference offers several key benefits:

Real-time visibility – Integration of DynamoDB and EventBridge provides real-time visibility into the status of batch inference jobs, enabling proactive monitoring and timely decision-making
Streamlined operations – Automated job monitoring and management minimizes manual overhead, reducing operational complexities so teams can focus on higher-value tasks like analyzing recommendation results
Optimized resource allocation – Metrics and insights about token count and latency stored in DynamoDB help organizations optimize resource allocation, facilitating efficient utilization of batch inference capabilities and cost-effectiveness

Prerequisites
To implement this solution, you must have the following:

An active AWS account with appropriate permissions to create resources, including S3 buckets, Lambda functions, and Amazon Bedrock resources.
Access to your selected models hosted on Amazon Bedrock. Make sure the selected model has been enabled in Amazon Bedrock.

Additionally, make sure to deploy the solution in an AWS Region that supports batch inference.
Deploy solution
For this solution, we provide an AWS CloudFormation template that sets up the services included in the architecture, to enable repeatable deployments. This template creates the following resources:

An S3 bucket to store the input and output
AWS Identity and Access Management (IAM) roles for Lambda functions, EventBridge rule, and Amazon Bedrock batch inference job
Amazon Bedrock Prompt Management template
EventBridge rule to trigger the Lambda function
DynamoDB table to store the job execution details

To deploy the CloudFormation template, complete the following steps:

Sign in to the AWS Management Console.
Open the template directly on the Create stack page of the CloudFormation console.
Choose Next and provide the following details:

For Stack name, enter a unique name.
For ModelId, enter the model ID that you need your batch job to run with. Only Anthropic Claude family models can be used with the CloudFormation template provided in this post.

Add optional tags, permissions, and other advanced settings if needed.
Review the stack details, select I acknowledge that AWS CloudFormation might create AWS IAM resources, and choose Next.
Choose Submit to initiate the deployment in your AWS account. The stack might take several minutes to complete.

Choose the Resources tab to find the newly created S3 bucket after the deployment succeeds.
Open the S3 bucket and confirm that there are two CSV files in your data folder.

On the Amazon S3 console, go to the data folder and create two more folders manually. This will prepare your S3 bucket to store the prompts and batch inference job results.

On the Lambda console, choose Functions in the navigation pane.
Choose the function that has create-jsonl-file in its name.

On the Test tab, choose Test to run the Lambda function. The function reads the CSV files from the S3 bucket and the prompt template, and creates a JSONL file with prompts for the customers under the prompts folder of your S3 bucket. The JSONL file has 100 prompts using the customers and products data. Lastly, the function submits a batch inference job with the CreateModelInvocationJob API call using the JSONL file.
On the Amazon Bedrock console, choose Prompt Management under Builder tools in the navigation pane.
Choose the finance-product-recommender-v1 prompt to see the prompt template input for the FM.
Choose Batch inference in the navigation pane under Inference and Assessment to find the submitted job.

The job progresses through different statuses: Submitted, Validating, In Progress, and lastly Completed, or Failed. You can leave this page and check the status after a few hours.
The EventBridge rule will automatically trigger the second Lambda function with event-bridge-trigger in its name on completion of the job. This function will add an entry in the DynamoDB table named bedrock_batch_job_status with details of the execution, as shown in the following screenshot.

This DynamoDB table functions as a state manager for Amazon Bedrock batch inference jobs, tracking the lifecycle of each request. The columns of the table are logically divided into the following categories:

Job identification and core attributes (job_arn, job_name) – These columns provide the unique identifier and a human-readable name for each batch inference request, serving as the primary keys or core attributes for tracking.
Execution and lifecycle management (StartTime, EndTime, last_processed_timestamp, TotalDuration) – This category captures the temporal aspects and the overall progression of the job, allowing for monitoring of its current state, start/end times, and total processing duration. last_processed_timestamp is crucial for understanding the most recent activity or checkpoint.
Processing statistics and performance (TotalRecordCount, ProcessedRecordCount, SuccessRecordCount, ErrorRecordCount) – These metrics provide granular insights into the processing efficiency and outcome of the batch job, highlighting data volume, successful processing rates, and error occurrences.
Cost and resource utilization metrics (InputTokenCount, OutputTokenCount) – Specifically designed for cost analysis, these columns track the consumption of tokens, which is a direct factor in Amazon Bedrock pricing, enabling accurate resource usage assessment.
Data and location management (InputLocation, OutputLocation) – These columns link the inference job to its source and destination data within Amazon S3, maintaining traceability of the data involved in the batch processing.

View product recommendations
Complete the following steps to open the output file and view the recommendations for each customer generated by the FM:

On the Amazon Bedrock console, open the completed batch inference job.
Find the job Amazon Resource Name (ARN) and copy the text after model-invocation-job/, as illustrated in the following screenshot.

Choose the link for S3 location under Output data. A new tab opens with the inference_results folder of the S3 bucket.

Search for the job results folder using the text copied from the previous step.
Open the folder to find two output files:

The file named manifest contains information like number of tokens, number of successful records, and number of errors.
The second output file contains the recommendations.

Download the second output file and open it in a text editor like Visual Studio Code to find the recommendations against each customer.

The example in the following screenshot shows several recommended products and why the FM chose this product for the specific customer.

Best practices
To optimize or enhance your monitoring solution, consider the following best practices:

Set up Amazon CloudWatch alarms for failed jobs to facilitate prompt attention to issues. For more details, see Amazon CloudWatch alarms.
Use appropriate DynamoDB capacity modes based on your workload patterns.
Configure relevant metrics and logging of batch job performance for operational visibility. Refer to Publish custom metrics for more details. The following are some useful metrics:

Average job duration
Token throughput rate (inputTokenCount + outputTokenCount) / jobDuration)
Error rates and types

Estimated costs
The cost estimate of running this solution one time is less than $1. The estimate for batch inference jobs considers Anthropic’s Claude 3.5 sonnet V2 model. Refer to Model pricing details for batch job pricing of other models on Amazon Bedrock.
Clean up
If you no longer need this automated monitoring solution, follow these steps to delete the resources it created to avoid additional costs:

On the Amazon S3 console, choose Buckets in the navigation pane.
Select the bucket you created and choose Empty to delete its contents.
On the AWS CloudFormation console, choose Stacks in the navigation pane.
Select the created stack and choose Delete.

This automatically deletes the deployed stack and the resources created.
Conclusion
In this post, we demonstrated how a financial services company can use an FM to process large volumes of customer records and get specific data-driven product recommendations. We also showed how to implement an automated monitoring solution for Amazon Bedrock batch inference jobs. By using EventBridge, Lambda, and DynamoDB, you can gain real-time visibility into batch processing operations, so you can efficiently generate personalized product recommendations based on customer credit data. The solution addresses key challenges in managing batch inference operations:

Alleviates the need for manual status checking or continuous polling
Provides immediate notifications when jobs complete or fail
Maintains a centralized record of job statuses

This automated monitoring approach significantly enhances the ability to process large amounts of financial data using batch inference for Amazon Bedrock. This solution offers a scalable, efficient, and cost-effective approach to do batch inference for a variety of use cases, such as generating product recommendations, identifying fraud patterns, or analyzing financial trends in bulk, with the added benefit of real-time operational visibility.

About the authors
Durga Prasad is a Senior Consultant at AWS, specializing in the Data and AI/ML. He has over 17 years of industry experience and is passionate about helping customers design, prototype, and scale Big Data and Generative AI applications using AWS native and open-source tech stacks.
Chanpreet Singh is a Senior Consultant at AWS with 18+ years of industry experience, specializing in Data Analytics and AI/ML solutions. He partners with enterprise customers to architect and implement cutting-edge solutions in Big Data, Machine Learning, and Generative AI using AWS native services, partner solutions and open-source technologies. A passionate technologist and problem solver, he balances his professional life with nature exploration, reading, and quality family time.

OpenAI has released AgentKit, a cohesive platform that packages a visual Agent Builder, an embeddable ChatKit UI, and expanded Evals into a single workflow for shipping production agents. The launch includes Agent Builder in beta and the rest generally available.

What’s new?

Agent Builder (beta). A visual canvas for composing multi-step, multi-agent workflows with drag-and-drop nodes, connectors, per-node guardrails, preview runs, inline eval configuration, and full versioning. Teams can start from templates or a blank canvas; the Responses API powers execution. OpenAI highlights internal and customer usage to compress iteration cycles when moving from prototype to production.

With Agent Builder, you can drag and drop nodes, connect tools, and publish your agentic workflows with ChatKit and the Agents SDK.https://t.co/ayLhKaSPUFHere’s @christinaahuang to walk you through it: pic.twitter.com/iFczB31hAl— OpenAI Developers (@OpenAIDevs) October 6, 2025

Agents SDK. A code-first alternative to the canvas with type-safe libraries in Node, Python, and Go. OpenAI positions the SDK as faster to integrate than manual prompt-and-tool orchestration while sharing the same execution substrate (Responses API).

@Albertsons used AgentKit to build an agent.An associate can ask it to create a plan to improve ice cream sales. The agent looks at the full context — seasonality, historical trends, external factors — and gives a recommendation. pic.twitter.com/rak7G5qc5U— OpenAI Developers (@OpenAIDevs) October 6, 2025

ChatKit (GA). A drop-in, brand-customizable chat interface for deploying agentic experiences on the web or in apps. It handles streaming, threads, and “thinking” UIs; the marketing page shows organizations using it for support and internal assistants.

Built-in tools and connectors. Agent workflows can call web search, file search, image generation, code interpreter, “computer use,” and external connectors, including Model Context Protocol (MCP) servers—reducing glue code for common tasks.

Connector Registry (beta). Centralized admin governance across ChatGPT and the API for data sources such as Dropbox, Google Drive, SharePoint, Microsoft Teams, and third-party MCPs. Rollout begins for customers with the Global Admin Console.

Evals (GA) and optimization. New capabilities include datasets, trace grading for end-to-end workflow assessment, automated prompt optimization, and third-party model evaluation. OpenAI emphasizes continuous measurement to raise task accuracy.

Pricing and availability. OpenAI states ChatKit and the new Evals features are GA; Agent Builder is beta. All are included under standard API model pricing (i.e., pay for model/compute usage rather than separate SKUs).

How the pieces fit in the puzzle?

Design: Use Agent Builder to visually assemble agents and guardrails, or write agents with the Agents SDK against the Responses API.

Deploy: Embed with ChatKit to deliver a production chat surface without building a frontend from scratch.

Optimize: Instrument with Evals (datasets, trace grading, graders) and iterate prompts based on graded traces.

How safety is included?

OpenAI’s launch materials pair Agent Builder with guardrails (open-source, modular) that can detect jailbreaks, mask/flag PII, and enforce policies at the node/tool boundary. Admins govern connections and data flows through the Connector Registry spanning both ChatGPT and the API.

Our Comments

It is a consolidated stack: AgentKit packages a visual Agent Builder for graph-based workflows, an embeddable ChatKit UI, and an Agents SDK that sits on top of the Responses API; this reduces bespoke orchestration and frontend work while keeping evaluation in-loop via datasets and trace grading. Our assessment: the value is operational—versioned node graphs, built-in tools (web/file search, computer use), connector governance, and standardized eval hooks are production concerns that previously required custom infrastructure.

Introducing AgentKit—build, deploy, and optimize agentic workflows. ChatKit: Embeddable, customizable chat UI Agent Builder: WYSIWYG workflow creator Guardrails: Safety screening for inputs/outputs Evals: Datasets, trace grading, auto-prompt optimization pic.twitter.com/pGgNHKOvj3— OpenAI Developers (@OpenAIDevs) October 6, 2025

The post OpenAI Debuts Agent Builder and AgentKit: A Visual-First Stack for Building, Deploying, and Evaluating AI Agents appeared first on MarkTechPost.

Do curated, tool-grounded demonstrations build stronger software agents than broad piles of generic instruction data? A team of researchers from Shanghai Jiao Tong University and SII Generative AI Research Lab (GAIR) proposes LIMI (“Less Is More for Agency”), a supervised fine-tuning method that turns a base model into a capable software/research agent using 78 samples. LIMI scores 73.5% average on AgencyBench (FTFC 71.7, RC@3 74.2, SR@3 74.6), beating strong baselines (GLM-4.5 45.1, Qwen3-235B-A22B 27.5, Kimi-K2 24.1, DeepSeek-V3.1 11.9) and even surpassing variants trained on 10,000 samples—with 128× less data.

https://arxiv.org/pdf/2509.17567

What exactly is new?

Agency Efficiency Principle: LIMI state that agentic competence scales more with data quality/structure than raw sample count. The research team fine-tune GLM-4.5/GLM-4.5-Air on 78 long-horizon, tool-use trajectories (samples) and report large gains on AgencyBench and generalization suites (TAU2-bench, EvalPlus-HE/MBPP, DS-1000, SciCode).

Minimal but dense supervision. Each trajectory (~13k–152k tokens; ~42.4k avg.) captures complete multi-turn workflows—model reasoning, tool calls, and environment observations—collected in the SII-CLI execution environment. Tasks span “vibe coding” (interactive software development) and research workflows (search, analysis, experiment design).

https://arxiv.org/pdf/2509.17567

How does it work?

Base models: GLM-4.5 (355B) and GLM-4.5-Air (106B). Training uses the slime SFT framework with identical configs across comparisons (to isolate data effects).

Data construction: 60 real queries from practitioners + 18 synthesized from high-star GitHub PRs (tight QA by PhD annotators). For each query, LIMI logs the full agent trajectory to successful completion inside SII-CLI.

Evaluation: AgencyBench (R=3 rounds) with FTFC, SR@3, RC@3; plus generalization suites (TAU2-airline/retail Pass^4, EvalPlus HE/MBPP, DS-1000, SciCode).

https://arxiv.org/pdf/2509.17567

Results

AgencyBench (avg): 73.5%. LIMI vs. GLM-4.5 (+28.4 pts); FTFC 71.7% vs 37.8%; SR@3 74.6% vs 47.4%.

Data efficiency: LIMI (78 samples) outperforms GLM-4.5 trained on AFM-CodeAgent SFT (10,000 samples): 73.5% vs 47.8%—+53.7% absolute with 128× less data. Similar gaps hold vs AFM-WebAgent (7,610) and CC-Bench-Traj (260).

Generalization: Across tool-use/coding/scientific computing, LIMI averages ~57%, exceeding GLM-4.5 and other baselines; without tool access, LIMI still leads slightly (50.0% vs 48.7% for GLM-4.5), indicating intrinsic gains beyond environment tooling.

https://arxiv.org/pdf/2509.17567

Key Takeaways

Data efficiency dominates scale. LIMI reaches 73.5% average on AgencyBench using curated trajectories, surpassing GLM-4.5 (45.1%) and showing a +53.7-point advantage over a 10k-sample SFT baseline—with 128× fewer samples.

Trajectory quality, not bulk. Training data are long-horizon, tool-grounded workflows in collaborative software development and scientific research, collected via the SII-CLI execution stack referenced by the paper.

Across-metric gains. On AgencyBench, LIMI reports FTFC 71.7%, SR@3 74.6%, and strong RC@3, with detailed tables showing large margins over baselines; generalization suites (TAU2, EvalPlus-HE/MBPP, DS-1000, SciCode) average 57.2%.

Works across scales. Fine-tuning GLM-4.5 (355B) and GLM-4.5-Air (106B) both yields large deltas over their bases, indicating method robustness to model size.

Our Comments

The research team trains GLM-4.5 variants with 78 curated, long-horizon, tool-grounded trajectories captured in a CLI environment spanning software-engineering and research tasks. It reports 73.5% average on AgencyBench with FTFC, RC@3, and SR@3 metrics; baseline GLM-4.5 is reported at 45.1%. A comparison against a 10,000-sample AFM-CodeAgent SFT baseline shows 73.5% vs 47.8%; tool-free evaluation indicates intrinsic gains (≈50.0% for LIMI vs 48.7% GLM-4.5). Trajectories are multi-turn and token-dense, emphasizing planning, tool orchestration, and verification.

Check out the Paper, GitHub Page and Model Card on HF. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter.
The post A New Agency-Focused Supervision Approach Scales Software AI Agents With Only 78 Examples appeared first on MarkTechPost.

Why treat LLM inference as batched kernels to DRAM when a dataflow compiler can pipe tiles through on-chip FIFOs and stream converters?StreamTensor is a compiler that lowers PyTorch LLM graphs (GPT-2, Llama, Qwen, Gemma) into stream-scheduled dataflow accelerators on AMD’s Alveo U55C FPGA. The system introduces an iterative tensor (“itensor”) type to encode tile/order of streams, enabling provably correct inter-kernel streaming and automated insertion/sizing of DMA engines, FIFOs, and layout converters. On LLM decoding workloads, the research team reports up to 0.64× lower latency vs. GPUs and up to 1.99× higher energy efficiency.

https://arxiv.org/pdf/2509.13694

What StreamTensor does?

StreamTensor compiles PyTorch graphs into a stream-oriented dataflow design so that intermediate tiles are largely avoids off-chip DRAM round-trips via on-chip streaming and fusion; DMAs are inserted only when required; they are forwarded through on-chip FIFOs to downstream kernels. The compiler’s central abstraction—iterative tensors (itensors)—records iteration order, tiling, and layout, which makes inter-kernel stream compatibility explicit and drives converter generation only where needed. The framework also searches hierarchically over tiling, fusion, and resource allocation, and uses a linear program to size FIFOs to avoid stalls or deadlock while minimizing on-chip memory.

https://arxiv.org/pdf/2509.13694

What’s actually new?

Hierarchical DSE. The compiler explores three design spaces—(i) tiling/unroll/vectorization/permutation at the Linalg level, (ii) fusion under memory/resource constraints, and (iii) resource allocation/stream widths—optimizing for sustained throughput under bandwidth limits.

End-to-end PyTorch → device flow. Models enter via Torch-MLIR, are transformed to MLIR Linalg, and then into a dataflow IR whose nodes become hardware kernels with explicit streams and host/runtime glue—no manual RTL assembly.

iterative tensor (itensor) typing system. A first-class tensor type expresses iteration order, tiling, and affine maps. This makes stream order explicit, allows safe kernel fusion, and lets the compiler synthesize minimal buffer/format converters when producers/consumers disagree.

Formal FIFO sizing. Inter-kernel buffering is solved with a linear-programming formulation to avoid stalls/deadlocks while minimizing on-chip memory usage (BRAM/URAM).

Results

Latency: up to 0.76× vs prior FPGA LLM accelerators and 0.64× vs a GPU baseline on GPT-2; Energy efficiency: up to 1.99× vs A100 on emerging LLMs (model-dependent). Platform context: Alveo U55C (HBM2 16 GB, 460 GB/s, PCIe Gen3×16 or dual Gen4×8, 2×QSFP28).

https://arxiv.org/pdf/2509.13694

Our Comments

The useful contribution here is a PyTorch→Torch-MLIR→dataflow compiler that emits stream-scheduled kernels and a host/runtime for AMD’s Alveo U55C; the iterative tensor type plus linear-programming-based FIFO sizing enables safe inter-kernel streaming rather than DRAM round-trips. On reported LLM decoding benchmarks across GPT-2, Llama, Qwen, and Gemma, the research team show geometric-mean latency as low as 0.64× vs. a GPU baseline and energy efficiency up to 1.99×, with scope limited to decoding workloads. The hardware context is clear: Alveo U55C provides 16 GB HBM2 at 460 GB/s with dual QSFP28 and PCIe Gen3×16 or dual Gen4×8, which aligns with the streaming dataflow design.

Check out the Paper. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter.
The post StreamTensor: A PyTorch-to-Accelerator Compiler that Streams LLM Intermediates Across FPGA Dataflows appeared first on MarkTechPost.

This post is cowritten with Gayathri Rengarajan and Harshit Kumar Nyati from PowerSchool.
PowerSchool is a leading provider of cloud-based software for K-12 education, serving over 60 million students in more than 90 countries and over 18,000 customers, including more than 90 of the top 100 districts by student enrollment in the United States. When we launched PowerBuddy, our AI assistant integrated across our multiple educational platforms, we faced a critical challenge: implementing content filtering sophisticated enough to distinguish between legitimate academic discussions and harmful content in educational contexts.
In this post, we demonstrate how we built and deployed a custom content filtering solution using Amazon SageMaker AI that achieved better accuracy while maintaining low false positive rates. We walk through our technical approach to fine tuning Llama 3.1 8B, our deployment architecture, and the performance results from internal validations.
PowerSchool’s PowerBuddy
PowerBuddy is an AI assistant that provides personalized insights, fosters engagement, and provides support throughout the educational journey. Educational leaders benefit from PowerBuddy being brought to their data and their users’ most common workflows within the PowerSchool ecosystem – such as Schoology Learning, Naviance CCLR, PowerSchool SIS, Performance Matters, and more – to ensure a consistent experience for students and their network of support providers at school and at home.
The PowerBuddy suite includes several AI solutions: PowerBuddy for Learning functions as a virtual tutor; PowerBuddy for College and Career provides insights for career exploration; PowerBuddy for Community simplifies access to district and school information, and others. The solution includes built-in accessibility features such as speech-to-text and text-to-speech functionality.
Content filtering for PowerBuddy
As an education technology provider serving millions of students—many of whom are minors—student safety is our highest priority. National data shows that approximately 20% of students ages 12–17 experience bullying, and 16% of high school students have reported seriously considering suicide. With PowerBuddy’s widespread adoption across K-12 schools, we needed robust guardrails specifically calibrated for educational environments.
The out-of-the-box content filtering and safety guardrails solutions available on the market didn’t fully meet PowerBuddy’s requirements, primarily because of the need for domain-specific awareness and fine-tuning within the education context. For example, when a high school student is learning about sensitive historical topics such as World War II or the Holocaust, it’s important that educational discussions aren’t mistakenly flagged for violent content. At the same time, the system must be able to detect and immediately alert school administrators to indications of potential harm or threats. Achieving this nuanced balance requires deep contextual understanding, which can only be enabled through targeted fine-tuning.
We needed to implement a sophisticated content filtering system that could intelligently differentiate between legitimate academic inquiries and truly harmful content—detecting and blocking prompts indicating bullying, self-harm, hate speech, inappropriate sexual content, violence, or harmful material not suitable for educational settings. Our challenge was finding a cloud solution to train and host a custom model that could reliably protect students while maintaining the educational functionality of PowerBuddy.
After evaluating multiple AI providers and cloud services that allow model customization and fine-tuning, we selected Amazon SageMaker AI as the most suitable platform based on these critical requirements:

Platform stability: As a mission-critical service supporting millions of students daily, we require an enterprise-grade infrastructure with high availability and reliability.
Autoscaling capabilities: Student usage patterns in education are highly cyclical, with significant traffic spikes during school hours. Our solution needed to handle these fluctuations without degrading performance.
Control of model weights after fine-tuning: We needed control over our fine-tuned models to enable continuous refinement of our safety guardrails, enabling us to quickly respond to new types of harmful content that might emerge in educational settings.
Incremental training capability: The ability to continually improve our content filtering model with new examples of problematic content was essential.
Cost-effectiveness: We needed a solution that would allow us to protect students without creating prohibitive costs that would limit schools’ access to our educational tools.
Granular control and transparency: Student safety demands visibility into how our filtering decisions are made, requiring a solution that isn’t a black box but provides transparency into model behavior and performance.
Mature managed service: Our team needed to focus on educational applications rather than infrastructure management, making a comprehensive managed service with production-ready capabilities essential.

Solution overview

Our content filtering system architecture, shown in the preceding figure, consists of several key components:

Data preparation pipeline:

Curated datasets of safe and unsafe content examples specific to educational contexts
Data preprocessing and augmentation to ensure robust model training
Secure storage in Amazon S3 buckets with appropriate encryption and access controls Note: All training data was fully anonymized and did not include personally identifiable student information

Model training infrastructure:

SageMaker training jobs for fine-tuning Llama 3.1 8B

Inference architecture:

Deployment on SageMaker managed endpoints with auto-scaling configured
Integration with PowerBuddy through Amazon API Gateway for real-time content filtering
Monitoring and logging through Amazon CloudWatch for continuous quality assessment

Continuous improvement loop:

Feedback collection mechanism for false positives/negatives
Scheduled retraining cycles to incorporate new data and improve performance
A/B testing framework to evaluate model improvements before full deployment

Development process
After exploring multiple approaches to content filtering, we decided to fine-tune Llama 3.1 8B using Amazon SageMaker JumpStart. This decision followed our initial attempts to develop a content filtering model from scratch, which proved challenging to optimize for consistency across various types of harmful content.
SageMaker JumpStart significantly accelerated our development process by providing pre-configured environments and optimized hyperparameters for fine-tuning foundation models. The platform’s streamlined workflow allowed our team to focus on curating high-quality training data specific to educational safety concerns rather than spending time on infrastructure setup and hyperparameter tuning.
We fine-tuned Llama 3.1 8B model using Low Rank Adaptation (LoRA) technique on Amazon SageMaker AI training jobs, which allowed us to maintain full control over the training process.
After the fine-tuning was done, we deployed the model on SageMaker AI managed endpoint and integrated it as a critical safety component within our PowerBuddy architecture.
For our production deployment, we selected NVIDIA A10G GPUs available through ml.g5.12xlarge instances, which offered the ideal balance of performance and cost-effectiveness for our model size. The AWS team provided crucial guidance on selecting optimal model serving configuration for our use case. This advice helped us optimize both performance and cost by ensuring we weren’t over-provisioning resources.
Technical implementation
Below is the code snippet to fine-tune the model on the pre-processed dataset. Instruction tuning dataset is first converted into domain adaptation dataset format and scripts utilize Fully Sharded Data Parallel (FSDP) as well as Low Rank Adaptation (LoRA) method for fine-tuning the model.
We define an estimator object first. By default, these models train via domain adaptation, so you must indicate instruction tuning by setting the instruction_tuned hyperparameter to True.

estimator = JumpStartEstimator(
model_id=model_id,
environment={“accept_eula”: “true”},
disable_output_compression=True,
hyperparameters={
“instruction_tuned”: “True”,
“epoch”: “5”,
“max_input_length”: “1024”,
“chat_dataset”: “False”
},
sagemaker_session=session,
base_job_name = “CF-M-0219251”
)

After we define the estimator, we are ready to start training:
estimator.fit({“training”: train_data_location})
After training, we created a model using the artifacts stored in S3 and deployed the model to a real-time endpoint for evaluation. We tested the model using our test dataset that covers key scenarios to validate performance and behavior. We calculated recall, F1, confusion matrix and inspected misclassifications. If needed, adjust hyperparameters/prompt template and retrain; otherwise proceed with production deployment.
You can also check out the sample notebook for fine tuning Llama 3 models on SageMaker JumpStart in SageMaker examples.
We used the Faster autoscaling on Amazon SageMaker realtime endpoints notebook to set up autoscaling on SageMaker AI endpoints.
Validation of solution
To validate our content filtering solution, we conducted extensive testing across multiple dimensions:

Accuracy testing: In our internal validation testing, the model achieved ~93% accuracy in identifying harmful content across a diverse test set representing various forms of inappropriate material.
False positive analysis: We worked to minimize instances where legitimate educational content was incorrectly flagged as harmful, achieving a false positive rate of less than 3.75% in test environments; results may vary by school context.
Performance testing: Our solution maintained response times averaging 1.5 seconds. Even during peak usage periods simulating real classroom environments, the system consistently delivered seamless user experience with no failed transactions.
Scalability and reliability validation:

Comprehensive load testing achieved 100% transaction success rate with consistent performance distribution, validating system reliability under sustained educational workload conditions.
Transactions completed successfully without degradation in performance or accuracy, demonstrating the system’s ability to scale effectively for classroom-sized concurrent usage scenarios.

Production deployment: Initial rollout to a select group of schools showed consistent performance in real-world educational environments.
Student safety outcomes: Schools reported a significant reduction in reported incidents of AI-enabled bullying or inappropriate content generation compared to other AI systems without specialized content filtering.

Fine-tuned model metrics compared to out-of-the-box content filtering solutions
The fine-tuned content filtering model demonstrated higher performance than generic, out-of-the-box filtering solutions in key safety metrics. It achieved a higher accuracy (0.93 compared to 0.89), and better F1-scores for both the safe (0.95 compared to 0.91) and unsafe (0.90 compared to 0.87) classes. The fine-tuned model also demonstrated a more balanced trade-off between precision and recall, indicating more consistent performance across classes. Importantly, it makes fewer false positive errors by misclassifying only 6 safe cases as unsafe, compared to 19 original responses in a test set of 160— a significant advantage in safety-sensitive applications. Overall, our fine-tuned content filtering model proved to be more reliable and effective.
Future plans
As the PowerBuddy suite evolves and is integrated into other PowerSchool products and agent flows, the content filter model will be continuously adapted and improved with fine tuning for other products with specific needs.
We plan to implement additional specialized adapters using the SageMaker AI multi-adapter inference feature alongside our content filtering model subject to feasibility and compliance consideration. The idea is to deploy fine-tuned small language models (SLMs) for specific problem solving in cases where large language models (LLMs) are huge and generic and don’t meet the need for narrower problem domains. For example:

Decision making agents specific to the Education domain
Data domain identification in cases of text to SQL queries

This approach will deliver significant cost savings by eliminating the need for separate model deployments while maintaining the specialized performance of each adapter.
The goal is to create an AI learning environment that is not only safe but also inclusive and responsive to diverse student needs across our global implementations, ultimately empowering students to learn effectively while being protected from harmful content.
Conclusion
The implementation of our specialized content filtering system on Amazon SageMaker AI has been transformative for PowerSchool’s ability to deliver safe AI experiences in educational settings. By building robust guardrails, we’ve addressed one of the primary concerns educators and parents have about introducing AI into classrooms—helping to ensure student safety.
As Shivani Stumpf, our Chief Product Officer, explains: “We’re now tracking around 500 school districts who’ve either purchased PowerBuddy or activated included features, reaching over 4.2 million students approximately. Our content filtering technology ensures students can benefit from AI-powered learning support without exposure to harmful content, creating a safe space for academic growth and exploration.”
The impact extends beyond just blocking harmful content. By establishing trust in our AI systems, we’ve enabled schools to embrace PowerBuddy as a valuable educational tool. Teachers report spending less time monitoring student interactions with technology and more time on personalized instruction. Students benefit from 24/7 learning support without the risks that might otherwise come with AI access.
For organizations requiring domain-specific safety guardrails, consider how the fine-tuning capabilities and managed endpoints of SageMaker AI can be adapted to your use case.
As we continue to expand PowerBuddy’s capabilities with the multi-adapter inference of SageMaker, we remain committed to maintaining the perfect balance between educational innovation and student safety—helping to ensure that AI becomes a positive force in education that parents, teachers, and students can trust.

About the authors
Gayathri Rengarajan is the Associate Director of Data Science at PowerSchool, leading the PowerBuddy initiative. Known for bridging deep technical expertise with strategic business needs, Gayathri has a proven track record of delivering enterprise-grade generative AI solutions from concept to production.
Harshit Kumar Nyati is a Lead Software Engineer at PowerSchool with 10+ years of experience in software engineering and analytics. He specializes in building enterprise-grade Generative AI applications using Amazon SageMaker AI, Amazon Bedrock, and other cloud services. His expertise includes fine-tuning LLMs, training ML models, hosting them in production, and designing MLOps pipelines to support the full lifecycle of AI applications.
Anjali Vijayakumar is a Senior Solutions Architect at AWS with over 9 years of experience helping customers build reliable and scalable cloud solutions. Based in Seattle, she specializes in architectural guidance for EdTech solutions, working closely with Education Technology companies to transform learning experiences through cloud innovation. Outside of work, Anjali enjoys exploring the Pacific Northwest through hiking.
Dmitry Soldatkin is a Senior AI/ML Solutions Architect at Amazon Web Services (AWS), helping customers design and build AI/ML solutions. Dmitry’s work covers a wide range of ML use cases, with a primary interest in Generative AI, deep learning, and scaling ML across the enterprise. He has helped companies in many industries, including insurance, financial services, utilities, and telecommunications. You can connect with Dmitry on LinkedIn.
Karan Jain is a Senior Machine Learning Specialist at AWS, where he leads the worldwide Go-To-Market strategy for Amazon SageMaker Inference. He helps customers accelerate their generative AI and ML journey on AWS by providing guidance on deployment, cost-optimization, and GTM strategy. He has led product, marketing, and business development efforts across industries for over 10 years, and is passionate about mapping complex service features to customer solutions.

Salesforce AI Research released CoDA-1.7B, a diffusion-based language model for code that generates by denoising whole sequences with bidirectional context, updating multiple tokens in parallel rather than left-to-right next-token prediction. The research team published both Base and Instruct checkpoints and an end-to-end training/evaluation/serving stack.

Understanding the architecture and training

CoDA adapts a 1.7B-parameter backbone to discrete diffusion for text: masked sequences are iteratively denoised using full-sequence attention, enabling native infilling and non-autoregressive decoding. The model card documents a three-stage pipeline (pre-training with bidirectional masking, supervised post-training, and progressive denoising at inference) plus reproducible scripts for TPU pre-training, GPU fine-tuning, and evaluation.

Key features surfaced in the release:

Bidirectional context via diffusion denoising (no fixed generation order).

Confidence-guided sampling (entropy-style decoding) to trade quality vs. speed.

Open training pipeline with deploy scripts and CLI.

How do they perform on Benchmarks?

On standard code-gen suites, CoDA-1.7B-Instruct reports: HumanEval 54.3%, HumanEval+ 47.6%, MBPP 47.2%, MBPP+ 63.2%, EvalPlus aggregate 55.4% (pass@1). For context, the model card compares against diffusion baselines including Dream-7B-Instruct (57.9% HumanEval), indicating CoDA’s 1.7B footprint is competitive with some 7B diffusion models on several metrics while using fewer parameters.

https://huggingface.co/Salesforce/CoDA-v0-Instruct

Inference behavior

Generation cost is governed by the number of diffusion steps; CoDA exposes knobs such as STEPS, ALG=”entropy”, ALG_TEMP, and block length to tune latency/quality trade-offs. Because tokens are updated in parallel under full attention, CoDA targets lower wall-clock latency at small scale compared with larger diffusion models, at comparable step budgets. (Hugging Face)

Deployment and licensing

The repository provides a FastAPI server with OpenAI-compatible APIs and an interactive CLI for local inference; instructions include environment setup and a start_server.sh launcher. Model cards and a Hugging Face collection centralize artifacts. The checkpoints are published under CC BY-NC 4.0 on Hugging Face.

Our Comments

CoDA-1.7B stands as a clean reference for discrete-diffusion code generation at small scale: 1.7B parameters, bidirectional denoising with parallel token updates, and a reproducible pipeline from pre-training to SFT and serving. The reported pass@1 results—HumanEval 54.3, HumanEval+ 47.6, MBPP 47.2, MBPP+ 63.2, EvalPlus aggregate 55.4—place it competitive with some 7B diffusion baselines (e.g., Dream-7B HumanEval 57.9) while using fewer parameters. Inference latency is explicitly governed by step count and decoding knobs (STEPS, entropy-style guidance), which is operationally useful for tuning throughput/quality. The release includes weights on Hugging Face and a FastAPI server/CLI for local deployment.

Check out the Paper, GitHub Repo and Model on Hugging Face. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post Salesforce AI Research Releases CoDA-1.7B: a Discrete-Diffusion Code Model with Bidirectional, Parallel Token Generation appeared first on MarkTechPost.

Building robust AI agents differs fundamentally from traditional software development, as it centers on probabilistic model behavior rather than deterministic code execution. This guide provides a neutral overview of methodologies for designing AI agents that are both reliable and adaptable, with an emphasis on creating clear boundaries, effective behaviors, and safe interactions.

What Is Agentic Design?

Agentic design refers to constructing AI systems capable of independent action within defined parameters. Unlike conventional coding, which specifies exact outcomes for inputs, agentic systems require designers to articulate desirable behaviors and trust the model to navigate specifics.

Variability in AI Responses

Traditional software outputs remain constant for identical inputs. In contrast, agentic systems—based on probabilistic models—produce varied yet contextually appropriate responses each time. This makes effective prompt and guideline design critical for both human-likeness and safety.

In an agentic system, a request like “Can you help me reset my password?” might elicit different yet appropriate replies such as “Of course! Please tell me your username,” “Absolutely, let’s get started—what’s your email address?” or “I can assist with that. Do you remember your account ID?”. This variability is purposeful, designed to enhance user experience by mimicking the nuance and flexibility of human dialogue. At the same time, this unpredictability requires thoughtful guidelines and safeguards so the system responds safely and consistently across scenarios

Why Clear Instructions Matter

Language models interpret instructions rather than execute them literally. Vague guidance such as:

Copy CodeCopiedUse a different Browseragent.create_guideline(
condition=”User expresses frustration”,
action=”Try to make them happy”
)

can lead to unpredictable or unsafe behavior, like unintended offers or promises. Instead, instructions should be concrete and action-focused:

Instead, be specific and safe:

Copy CodeCopiedUse a different Browseragent.create_guideline(
condition=”User is upset by a delayed delivery”,
action=”Acknowledge the delay, apologize, and provide a status update”
)

This approach ensures the model’s actions align with organizational policy and user expectations.

Building Compliance: Layers of Control

LLMs can’t be fully “controlled,” but you can still guide and constrain their behavior effectively.

Layer 1: Guidelines

Use guidelines to define and shape normal behavior.

Copy CodeCopiedUse a different Browserawait agent.create_guideline(
condition=”Customer asks about topics outside your scope”,
action=”Politely decline and redirect to what you can help with”
)

Layer 2: Canned Responses

For high-risk situations (such as policy or medical advice), use pre-approved canned responses to ensure consistency and safety.

Copy CodeCopiedUse a different Browserawait agent.create_canned_response(
template=”I can help with account questions, but for policy details I’ll connect you to a specialist.”
)

This layered approach minimizes risk and ensures the agent never improvises in sensitive situations.

Tool Calling: When Agents Take Action

When AI agents take action using tools such as APIs or functions, the process involves more complexity than simply executing a command. For example, if a user says, “Schedule a meeting with Sarah for next week,” the agent must interpret several unclear elements: Which Sarah is being referred to? What specific day and time within “next week” should the meeting be scheduled? And on which calendar?

This illustrates the Parameter Guessing Problem, where the agent attempts to infer missing details that weren’t explicitly provided. To address this, tools should be designed with clear purpose descriptions, parameter hints, and contextual examples to reduce ambiguity. Additionally, tool names should be intuitive and parameter types consistent, helping the agent reliably select and populate inputs. Well-structured tools improve accuracy, reduce errors, and make the interactions smoother and more predictable for both the agent and the user.

This thoughtful tool design practice is essential for effective, safe agent functionality in real-world applications.When AI agents perform tasks through tools such as APIs or functions, the complexity is often higher than it initially appears.

Agent Design Is Iterative

Unlike static software, agent behavior in agentic systems is not fixed; it matures over time through a continuous cycle of observation, evaluation, and refinement. The process typically begins with implementing straightforward, high-frequency user scenarios—those “happy path” interactions where the agent’s responses can be easily anticipated and validated. Once deployed in a safe testing environment, the agent’s behavior is closely monitored for unexpected answers, user confusion, or any breaches of policy guidelines.

As issues are observed, the agent is systematically improved by introducing targeted rules or refining existing logic to address problematic cases. For example, if users repeatedly decline an upsell offer but the agent continues to bring it up, a focused rule can be added to prevent this behavior within the same session. Through this deliberate, incremental tuning, the agent gradually evolves from a basic prototype into a sophisticated conversational system that is responsive, reliable, and well-aligned with both user expectations and operational constraints.

Writing Effective Guidelines

Each guideline has three key parts:

Example:

Copy CodeCopiedUse a different Browserawait agent.create_guideline(
condition=”Customer requests a specific appointment time that’s unavailable”,
action=”Offer the three closest available slots as alternatives”,
tools=[get_available_slots]
)

Structured Conversations: Journeys

For complex tasks such as booking appointments, onboarding, or troubleshooting, simple guidelines alone are often insufficient. This is where Journeys become essential. Journeys provide a framework to design structured, multi-step conversational flows that guide the user through a process smoothly while maintaining a natural dialogue.

For example, a booking flow can be initiated by creating a journey with a clear title and conditions defining when it applies, such as when a customer wants to schedule an appointment. The journey then progresses through states—first asking the customer what type of service they need, then checking availability using an appropriate tool, and finally offering available time slots. This structured approach balances flexibility and control, enabling the agent to handle complex interactions efficiently without losing the conversational feel.

Example: Booking Flow

Copy CodeCopiedUse a different Browserbooking_journey = await agent.create_journey(
title=”Book Appointment”,
conditions=[“Customer wants to schedule an appointment”],
description=”Guide customer through the booking process”
)

t1 = await booking_journey.initial_state.transition_to(
chat_state=”Ask what type of service they need”
)
t2 = await t1.target.transition_to(
tool_state=check_availability_for_service
)
t3 = await t2.target.transition_to(
chat_state=”Offer available time slots”
)

Balancing Flexibility and Predictability

Balancing flexibility and predictability is essential when designing an AI agent. The agent should feel natural and conversational, rather than overly scripted, but it must still operate within safe and consistent boundaries. 

If instructions are too rigid—for example, telling the agent to “Say exactly: ‘Our premium plan is $99/month‘”—the interaction can feel mechanical and unnatural. On the other hand, instructions that are too vague, such as “Help them understand our pricing“, can lead to unpredictable or inconsistent responses. 

A balanced approach provides clear direction while allowing the agent some adaptability, for example: “Explain our pricing tiers clearly, highlight the value, and ask about the customer’s needs to recommend the best fit.” This ensures the agent remains both reliable and engaging in its interactions.

Designing for Real Conversations

Designing for real conversations requires recognizing that, unlike web forms, conversations are non-linear. Users may change their minds, skip steps, or move the discussion in unexpected directions. To handle this effectively, there are several key principles to follow. 

Context preservation ensures the agent keeps track of information already provided so it can respond appropriately. 

Progressive disclosure means revealing options or information gradually, rather than overwhelming the user with everything at once. 

Recovery mechanisms allow the agent to manage misunderstandings or deviations gracefully, for example by rephrasing a response or gently redirecting the conversation for clarity. 

This approach helps create interactions that feel natural, flexible, and user-friendly.

Effective agentic design means starting with core features, focusing on main tasks before tackling rare cases. It involves careful monitoring to spot any issues in the agent’s behavior. Improvements should be based on real observations, adding clear rules to guide better responses. It’s important to balance clear boundaries that keep the agent safe while allowing natural, flexible conversation. For complex tasks, use structured flows called journeys to guide multi-step interactions. Finally, be transparent about what the agent can do and its limits to set proper expectations. This simple process helps create reliable, user-friendly AI agents.
The post Agentic Design Methodology: How to Build Reliable and Human-Like AI Agents using Parlant appeared first on MarkTechPost.