Category: Uncategorized

OpenAI has just introduced GPT-5.2, its most advanced frontier model for professional work and long running agents, and is rolling it out across ChatGPT and the API.

GPT-5.2 is a family of three variants. In ChatGPT, users see ChatGPT-5.2 Instant, Thinking and Pro. In the API, the corresponding models are gpt-5.2-chat-latest, gpt-5.2, and gpt-5.2-pro. Instant targets everyday assistance and learning, Thinking targets complex multi step work and agents, and Pro allocates more compute for hard technical and analytical tasks.

Benchmark profile, from GDPval to SWE Bench

GPT-5.2 Thinking is positioned as the main workhorse for real world knowledge work. On GDPval, an evaluation of well specified knowledge tasks across 44 occupations in 9 large industries, it beats or ties top industry professionals on 70.9 percent of comparisons, while producing outputs at more than 11 times the speed and under 1 percent of the estimated expert cost. For engineering teams this means the model can reliably generate artifacts such as presentations, spreadsheets, schedules, and diagrams given structured instructions.

On an internal benchmark of junior investment banking spreadsheet modeling tasks, average scores rise from 59.1 percent with GPT-5.1 to 68.4 percent with GPT-5.2 Thinking and 71.7 percent with GPT-5.2 Pro. These tasks include three statement models and leveraged buyout models with constraints on formatting and citations, which is representative of many structured enterprise workflows.

In software engineering, GPT-5.2 Thinking reaches 55.6 percent on SWE-Bench Pro and 80.0 percent on SWE-bench Verified. SWE-Bench Pro evaluates repository level patch generation over multiple languages, while SWE-bench Verified focuses on Python.

Long context and agentic workflows

Long context is a core design target. GPT-5.2 Thinking sets a new state of the art on OpenAI MRCRv2, a benchmark that inserts multiple identical ‘needle’ queries into long dialogue “haystacks” and measures whether the model can reproduce the correct answer. It is the first model reported to reach near 100 percent accuracy on the 4 needle MRCR variant out to 256k tokens.

For workloads that exceed even that context, GPT-5.2 Thinking integrates with the Responses /compact endpoint, which performs context compaction to extend the effective window for tool heavy, long running jobs. This is relevant if you are building agents that iteratively call tools over many steps and need to maintain state beyond the raw token limit.

On tool usage, GPT-5.2 Thinking reaches 98.7 percent on Tau2-bench Telecom, a multi turn customer support benchmark where the model must orchestrate tool calls across a realistic workflow. The official examples from OpenAI release post show scenarios like a traveler with a delayed flight, missed connection, lost bag and medical seating requirement, where GPT-5.2 manages rebooking, special assistance seating and compensation in a consistent sequence while GPT-5.1 leaves steps unfinished.

Vision, science and math

Vision quality also moves up. GPT-5.2 Thinking roughly halves error rates on chart reasoning and user interface understanding benchmarks like CharXiv Reasoning and ScreenSpot Pro when a Python tool is enabled. The model shows improved spatial understanding of images, for example when labeling motherboard components with approximate bounding boxes, GPT-5.2 identifies more regions with tighter placement than GPT-5.1.

For scientific workloads, GPT-5.2 Pro scores 93.2 percent and GPT-5.2 Thinking 92.4 percent on GPQA Diamond, and GPT-5.2 Thinking solves 40.3 percent of FrontierMath Tier 1 to Tier 3 problems with Python tools enabled. These benchmarks cover graduate level physics, chemistry, biology and expert mathematics, and OpenAI highlights early use where GPT-5.2 Pro contributed to a proof in statistical learning theory under human verification.

Comparison Table

ModelPrimary positioningContext window / max outputKnowledge cutoffNotable benchmarks (Thinking / Pro vs GPT-5.1 Thinking)GPT-5.1 Flagship model for coding and agentic tasks with configurable reasoning effort400,000 tokens context, 128,000 max output2024-09-30SWE-Bench Pro 50.8 percent, SWE-bench Verified 76.3 percent, ARC-AGI-1 72.8 percent, ARC-AGI-2 17.6 percentGPT-5.2 (Thinking) New flagship model for coding and agentic tasks across industries and for long running agents400,000 tokens context, 128,000 max output2025-08-31GDPval wins or ties 70.9 percent vs industry professionals, SWE-Bench Pro 55.6 percent, SWE-bench Verified 80.0 percent, ARC-AGI-1 86.2 percent, ARC-AGI-2 52.9 percentGPT-5.2 ProHigher compute version of GPT-5.2 for the hardest reasoning and scientific workloads, produces smarter and more precise responses400,000 tokens context, 128,000 max output2025-08-31GPQA Diamond 93.2 percent vs 92.4 percent for GPT-5.2 Thinking and 88.1 percent for GPT-5.1 Thinking, ARC-AGI-1 90.5 percent and ARC-AGI-2 54.2 percent

Key Takeaways

GPT-5.2 Thinking is the new default workhorse model: It replaces GPT-5.1 Thinking as the main model for coding, knowledge work and agents, while keeping the same 400k context and 128k max output, but with clearly higher benchmark performance across GDPval, SWE-Bench, ARC-AGI and scientific QA.

Substantial accuracy jump over GPT-5.1 at similar scale: On key benchmarks, GPT-5.2 Thinking moves from 50.8 percent to 55.6 percent on SWE-Bench Pro and from 76.3 percent to 80.0 percent on SWE-bench Verified, and from 72.8 percent to 86.2 percent on ARC-AGI-1 and from 17.6 percent to 52.9 percent on ARC-AGI-2, while keeping token limits comparable.

GPT-5.2 Pro is targeted at high end reasoning and science: GPT-5.2 Pro is a higher compute variant that mainly improves hard reasoning and scientific tasks, for example reaching 93.2 percent on GPQA Diamond versus 92.4 percent for GPT-5.2 Thinking and 88.1 percent for GPT-5.1 Thinking, and higher scores on ARC-AGI tiers.

The post OpenAI Introduces GPT 5.2: A Long Context Workhorse For Agents, Coding And Knowledge Work appeared first on MarkTechPost.

Los Angeles, December 11, 2025 — Marktechpost has released ML Global Impact Report 2025 (AIResearchTrends.com). This educational report’s analysis includes over 5,000 articles from more than 125 countries, all published within the Nature family of journals between January 1 and September 30, 2025. The scope of this report is strictly confined to this specific body of work and is not a comprehensive assessment of global research.This report focuses solely on the specific work presented and does not represent a full evaluation of worldwide research.

The ML Global Impact Report 2025 focuses on three core questions:

In which disciplines has ML become part of the standard methodological toolkit, and where is adoption still sparse.

Which kinds of problems are most likely to rely on ML, such as high-dimensional imaging, sequence data, or complex physical simulations.

How ML usage patterns differ by geography and research ecosystem, based on the global footprint of these selected 5,000 papers.

ML has most frequently become part of the standard methodological toolkit within the disciplines of applied sciences and health research, where it is often employed as a critical step within a larger experimental workflow rather than being the main subject of research itself. The analysis of the papers indicates that ML’s adoption is concentrated in these domains, with the tools serving to augment existing research pipelines. The report aims to distinguish these areas of common use from other fields where the integration of machine learning remains less frequent.

The kinds of problems most likely to rely on machine learning are those involving complex data analysis tasks, such as high-dimensional imaging, sequence data analysis, and intricate physical simulations. The report tracks the specific task types, including prediction, classification, segmentation, sequence modeling, feature extraction, and simulation, to understand where ML is being applied. This categorization highlights the utility of machine learning across different stages of the research process, from initial data processing to final output generation.

ML usage patterns show a distinct geographical separation between the origins of the tools and the heavy users of the technology. The majority of machine learning tools cited in the corpus originate from organizations based in the United States, which maintains many widely used frameworks and libraries. In contrast, China is identified as the largest contributor to the research papers, accounting for about 40% of all ML-tagged papers, significantly more than the United States’ contribution of around 18%. The report also highlights the global ecosystem by citing frequently used non-US tools, such as Scikit-learn (France), U-Net (Germany), and CatBoost (Russia), along with tools originated from Canada including GAN and RNN families.Overall, the ML Global Impact Report 2025 provides deep insights into the global research ecosystem, highlighting that Machine Learning has become a standard methodological tool primarily within applied sciences and health research. The analysis reveals a concentration of ML usage on complex data challenges, such as high-dimensional imaging and physical simulations. A core finding is the clear geographical split between the origin of ML tools—many maintained by US organizations—and the heaviest users of the technology, with China accounting for a significantly higher number of ML-tagged research papers in the analyzed corpus. These patterns are specific to the 5,000+ Nature family articles analysed, underscoring the report’s focused view on current research workflows.
The post The Machine Learning Divide: Marktechpost’s Latest ML Global Impact Report Reveals Geographic Asymmetry Between ML Tool Origins and Research Adoption appeared first on MarkTechPost.

This post was written with Arun Sittampalam and Maxime Darcot from Swisscom.
As we navigate the constantly shifting AI ecosystem, enterprises face challenges in translating AI’s potential into scalable, production-ready solutions. Swisscom, Switzerland’s leading telecommunications provider with an estimated $19B revenue (2025) and over $37B Market capitalization as of June 2025 exemplifies how organizations can successfully navigate this complexity while maintaining their commitment to sustainability and excellence.
Swisscom has been recognized as the Most Sustainable Company in the Telecom industry for 3 consecutive years by World Finance magazine, Swisscom has established itself as an innovation leader committed to achieving net-zero greenhouse gas emissions by 2035 in alignment with the Paris Climate Agreement. This sustainability-first approach extends to their AI strategy where they’re breaking through what they call the “automation ceiling” – where traditional automation approaches fail to meet modern business demands.
In this post, we’ll show how Swisscom implemented Amazon Bedrock AgentCore to build and scale their enterprise AI agents for customer support and sales operations. As an early adopter of Amazon Bedrock in the AWS Europe Region (Zurich), Swisscom leads in enterprise AI implementation with their Chatbot Builder system and various AI initiatives. Their successful deployments include Conversational AI powered by Rasa and fine-tuned LLMs on Amazon SageMaker, and the Swisscom Swisscom myAI assistant, built to meet Swiss data protection standards.
Solution overview: Swisscom’s agentic AI enabler framework
The challenge of enterprise-wide scaling of AI agents lies in managing siloed agentic solutions while facilitating cross-departmental coordination. Swisscom addresses this through Model Context Protocol (MCP) servers and the Agent2Agent protocol (A2A), for seamless agent communication across domains. Operating under Switzerland’s strict data protection laws, they’ve developed a framework that balances compliance requirements with efficient scaling capabilities, helping prevent redundant efforts while maintaining high security standards.
Swisscom’s multi-agent architecture: System design and implementation challenges
Swisscom’s vision for enterprise-level agentic AI focuses on addressing fundamental challenges that organizations face when scaling AI solutions. They recognise that successful implementation requires more than just innovative technology, it demands a comprehensive approach to infrastructure and operations. One of the key challenges lies in orchestrating AI agents across different departments and systems while maintaining security and efficiency.
To illustrate these challenges in practice, let’s examine a common customer service scenario where an agent is tasked with helping a customer restore their Internet router connectivity. There are three potential causes for the connectivity loss: 1) a billing issue, 2) a network outage, or 3) a configuration mismatch known as a pairing issue. These issues typically reside in departments different from where the assigned agent operates, highlighting the need for seamless cross-departmental coordination.
The architecture diagram below illustrates the vision and associated challenges for a generic customer agent without the Amazon Bedrock AgentCore. The shared VPC setup of Swisscom is explained in more detail in the blog post, Automated networking with shared VPCs at Swisscom.

This architecture includes the following components:

A customer-facing generic agent deployed as a containerized runtime within a shared VPC, requiring both foundation model invocation capabilities and robust session management.
For task completion, the agent requires to access to other agents and MCP servers. These resources are typically distributed across multiple AWS accounts and are deployed as containerised runtimes within the shared VPC.
Internal application access primarily occurs through SAIL (Service and Interface Library), Swisscom’s central system for API hosting and service integration. Corporate network resources are accessible via AWS Direct Connect, with a VPC Transit Gateway facilitating secure cross-network communication.
Security compliance is paramount: each interaction requires temporary access tokens that authenticate both the agent and the customer context. This bidirectional validation is essential to the system components – agents, MCP servers, and tools must verify incoming tokens for service requests.
Gaining long-term insights from the stored sessions, such as customer preferences, demands a sophisticated analysis.

To build the solution mentioned above at scale, Swisscom identified several critical challenges that needed to be addressed:

Security and Authentication:

How to implement secure, transitive authentication and authorization that enforces least-privilege access based on intersecting permissions (customer, agent, department)?
How to enable controlled resource sharing across departments, cloud systems, and on-premises networks?

Integration and Interoperability:

How to make MCP servers and other agents centrally available to other use cases?
How to integrate and maintain compatibility with existing agentic use cases across Swisscom’s infrastructure?

Customer Intelligence and Operations:

How to effectively capture and utilize customer insights across multiple agentic interactions?
How to implement standardized evaluation and observability practices across the agents?

How Amazon Bedrock AgentCore addresses the challenges
Amazon Bedrock AgentCore provides Swisscom with a comprehensive solution that addresses their enterprise-scale agentic AI challenges.

AgentCore Runtime: Enables Swisscom’s developers to focus on building agents while the system handles secure, cost-efficient hosting and automatic scaling through Docker container deployment that maintains session-level isolation. Hosted in the shared VPC allows access to internal APIs.
AgentCore Identity: Seamlessly integrates with Swisscom’s existing identity provider, managing both inbound and outbound authentication, alleviating the need for custom token exchange servers and simplifying secure interactions between agents, tools, and data sources.
AgentCore Memory: Delivers a robust solution for managing both session-based and long-term memory storage with custom memory strategies. This is particularly valuable for B2C operations where understanding customer context across interactions is crucial. Keeping each user’s data separate also supports security and compliance efforts.
Strands Agents Framework: Demonstrates high adoption among Swisscom’s developers due to its simplified agent construction, faster development cycles, seamless integration with Bedrock AgentCore services, and built-in capabilities for tracing, evaluation, and OpenTelemetry logging.

This solution does the following:

The client sends a request to the Strands agent running on AgentCore Runtime, passing an authentication token from the Swisscom IdP.
The client’s token is validated and a new token for the agent’s downstream tool usage is generated and passed back to the agent.
The agent invokes the foundation model on Bedrock and stores the sessions in the AgentCore Memory. The traffic traverses the VPC endpoints for Bedrock and Bedrock AgentCore, keeping the traffic private.
The agent accesses internal APIs, MCP & A2A servers inside the shared VPC, authenticating with the temporary token from AgentCore Identity.

With the flexibility to use a subset of features of Amazon Bedrock AgentCore and their Amazon VPC integration Swisscom could remain secure and flexible to use the Bedrock AgentCore services for their specific needs, for example to integrate with existing agents on Amazon EKS. Amazon Bedrock AgentCore integrates with VPC to facilitate secure communication between agents and internal resources.
Results and benefits: Real-world implementation with self-service use case
Swisscom partnered with AWS to implement Amazon Bedrock AgentCore for two B2C cases: 1) generating personalized sales pitches, and 2) providing automated customer support for technical issues like self-service troubleshooting. Both agents are being integrated into Swisscom’s existing customer generative AI-powered chatbot system called SAM, necessitating high-performance agent-to-agent communication protocols due to the high volume of Swisscom customers and strict latency requirements. Throughout the development process, the team created an agent for each use case designed to be shared across the organization through MCP and A2A.
Amazon Bedrock AgentCore has proven instrumental in these implementations. By using the Bedrock AgentCore Memory long-term insights Swisscom can track and analyze customer interactions across different touchpoints, continuously improving the customer experience across domains. AgentCore Identity facilitates robust security, implementing precise access controls that limit agents to only those resources authorized for the specific customer interaction. The scalability of AgentCore Runtime allows these agents to efficiently handle thousands of requests per month each, maintaining low latency while optimizing costs.
The adoption of Strands Agents framework has been particularly valuable in this journey:

Development teams achieved their first business stakeholder demos within 3-4 weeks, despite having no prior experience with Strands Agents.
One project team migrated from their LangGraph implementation to Strands Agents, citing reduced complexity and faster development cycles.
The framework’s native OpenTelemetry integration supported seamless export of performance traces to Swisscom’s existing observability infrastructure, maintaining consistency with enterprise-wide monitoring standards.
The Strands evaluation test cases allowed teams quickly put an evaluation pipeline together without the need of additional tools, for a quick validation of the PoC.

Conclusion: Enterprise AI at scale – Key insights and Strategic implications
Swisscom’s implementation of Amazon Bedrock AgentCore demonstrates how enterprises can successfully navigate the complexities of production-ready Agentic AI while maintaining regulatory compliance and operational excellence. Swisscom’s journey offers 3 critical insights:

Architectural foundation matters: By addressing the fundamental challenges of secure cross-org authentication, standardized agent orchestration, and comprehensive observability, Swisscom established a scalable foundation that accelerates deployment rather than constraining it. The integration of AgentCore Runtime, Identity, and Memory services accelerated the infrastructure setup so teams could focus on business value.
Framework selection drives velocity: The adoption of Strands Agents framework exemplifies how the right development tools can dramatically reduce time-to-value. Teams achieving stakeholder demos within 3-4 weeks, coupled with successful migrations from alternative frameworks, validates the importance of developer experience in enterprise AI adoption.
Compliance as an enabler: Swisscom proved that regulatory compliance need not impede innovation. The system’s ability to scale while maintaining data sovereignty and user privacy has proven particularly valuable in the Swiss industry, where regulatory compliance is paramount.

As enterprises increasingly recognize AI agents as fundamental to competitive advantage, Swisscom’s implementation provides a proven reference architecture. Their success with high-volume B2C applications—from personalized sales assistance to automated technical support—illustrates that agentic AI can deliver measurable business outcomes at scale when built on appropriate infrastructure. This implementation serves as a blueprint for organizations seeking to deploy enterprise-scale AI solutions, showing how careful architectural planning and the right technology choices can lead to successful outcomes in both customer service and sales operations.
Next steps and looking ahead
The future roadmap focuses on three key areas: agent sharing, cross-domain integration, and governance. A centralized agent registry will facilitate discovery and reuse across the organization, supported by standardized documentation and shared best practices. Cross-domain integration will enable seamless collaboration between different business units, with clear standards for agent communication and interoperability. The implementation of robust governance mechanisms, including version control, usage monitoring, and regular security audits, will facilitate sustainable growth of the system while maintaining compliance with enterprise standards. This comprehensive approach will help drive continuous improvement based on real-world usage patterns and feedback.
Check out these additional links for relevant Agentic related information:

Transforming network operations with AI: How Swisscom built a network assistant using Amazon Bedrock
Introducing Amazon Bedrock AgentCore: Securely deploy and operate AI agents at any scale
Amazon Bedrock AgentCore Runtime, Browser, and Code Interpreter add support for VPC, AWS PrivateLink, CloudFormation, and tagging
Secure ingress connectivity to Amazon Bedrock AgentCore Gateway using interface VPC endpoints

About the authors
Arun Sittampalam, Director of Product Management AI at Swisscom, leads the company’s transformation toward Agentic AI, designing frameworks that scale large language model (LLM)–driven agents across enterprise environments. His team is building Swisscom’s agentic platform, integrating Amazon Bedrock, AgentCore and internal orchestration frameworks to empower Swisscom’s AI product teams to build and scale intelligent agents faster. Arun focuses on operationalizing multi-agent architectures that deliver automation, reliability, and scalability.
Maxime is a System and Security Architect at Swisscom, responsible for the architecture of Conversational and Agentic AI enablement. He is originally a Data Scientist with 10 years of experience in developing, deploying and maintaining NLP solutions which have been helping millions of Swisscom customers.
Julian Grüber is a Data Science Consultant at Amazon Web Services. He partners with strategic customers to scale GenAI solutions that unlock business value, working at both the use case and enterprise architecture level. Drawing on his background in applied mathematics, machine learning, business, and cloud infrastructure, Julian bridges technical depth with business outcomes to address complex AI/ML challenges.
Marco Fischer is a Senior Solutions Architect at Amazon Web Services. He works with leading telecom operators to design and deploy scalable, production-ready solutions. With over two decades of experience spanning software engineering, architecture, and cloud infrastructure, Marco combines deep technical expertise with a passion for solving complex enterprise challenges.
Akarsha Sehwag is a Generative AI Data Scientist for Amazon Bedrock AgentCore GTM team. With over six years of expertise in AI/ML, she has built production-ready enterprise solutions across diverse customer segments in Generative AI, Deep Learning and Computer Vision domains. Outside of work, she likes to hike, bike or play Badminton.
Ruben Merz is a Principal Solutions Architect at AWS, specializing in digital sovereignty, AI, and networking solutions for enterprise customers. With deep expertise in distributed systems and networking, he architects secure, compliant cloud solutions that help organizations navigate complex regulatory requirements while accelerating their digital transformation journeys.

Today we’re announcing Amazon SageMaker AI with MLflow, now including a serverless capability that dynamically manages infrastructure provisioning, scaling, and operations for artificial intelligence and machine learning (AI/ML) development tasks. It scales resources up during intensive experimentation and down to zero when not in use, reducing operational overhead. It introduces enterprise-scale features including seamless access management with cross-account sharing, automated version upgrades, and integration with SageMaker AI capabilities like model customization and pipelines. With no administrator configuration needed and at no additional cost, data scientists can immediately begin tracking experiments, implementing observability, and evaluating model performance without infrastructure delays, making it straightforward to scale MLflow workloads across your organization while maintaining security and governance.
In this post, we explore how these new capabilities help you run large MLflow workloads—from generative AI agents to large language model (LLM) experimentation—with improved performance, automation, and security using SageMaker AI with MLflow.
Enterprise scale features in SageMaker AI with MLflow
The new MLflow serverless capability in SageMaker AI delivers enterprise-grade management with automatic scaling, default provisioning, seamless version upgrades, simplified AWS Identity and Access Management (IAM) authorization, resource sharing through AWS Resource Access Manager (AWS RAM), and integration with both Amazon SageMaker Pipelines and model customization. The term MLflow Apps replaces the previous MLflow tracking servers terminology, reflecting the simplified, application-focused approach. You can access the new MLflow Apps page in Amazon SageMaker Studio, as shown in the following screenshot.

A default MLflow App is automatically provisioned when you create a SageMaker Studio domain, streamlining the setup process. It’s enterprise-ready out of the box, requiring no additional provisioning or configuration. The MLflow App scales elastically with your usage, alleviating the need for manual capacity planning. Your training, tracking, and experimentation workloads can get the resources they need automatically, simplifying operations while maintaining performance.
Administrators can define a maintenance window during the creation of the MLflow App, during which in-place version upgrades of the MLflow App take place. This helps the MLflow App be standardized, secure, and continuously up to date, minimizing manual maintenance overhead. MLflow version 3.4 is supported with this launch, and as shown in the following screenshot, extends MLflow to ML, generative AI applications, and agent workloads.

Simplified identity management with MLflow Apps
We’ve simplified access control and IAM permissions for ML teams with the new MLflow App. A streamlined permissions set, such as sagemaker:CallMlflowAppApi, now covers common MLflow operations—from creating and searching experiments to updating trace information—making access control more straightforward to enforce.
By enabling simplified IAM permissions boundaries, users and platform administrators can standardize IAM roles across teams, personas, and projects, facilitating consistent and auditable access to MLflow experiments and metadata. For complete IAM permission and policy configurations, see Set up IAM permissions for MLflow Apps.
Cross-account sharing of MLflow Apps using AWS RAM
Administrators want to centrally manage their MLflow infrastructure while provisioning access across different AWS accounts. MLflow Apps support AWS cross-account sharing for collaborative enterprise AI development. Using AWS RAM, this feature helps AI platform administrators share an MLflow App seamlessly across data scientists with consumer AWS accounts, as illustrated in the following diagram.

Platform administrators can maintain a centralized, governed SageMaker domain that provisions and manages the MLflow App, and data scientists in separate consuming accounts can launch and interact with the MLflow App securely. Combined with the new simplified IAM permissions, enterprises can launch and manage an MLflow App from a centralized administrative AWS account. Using the shared MLflow App, a downstream data scientist consumer can log their MLflow experimentation and generative AI workloads while maintaining governance, auditability, and compliance from a single platform administrator control plane. To learn more about cross-account sharing, see Getting Started with AWS RAM.
SageMaker Pipelines and MLflow integration
SageMaker Pipelines is integrated with MLflow. SageMaker Pipelines is a serverless workflow orchestration service purpose-built for MLOps and LLMOps automation. You can seamlessly build, execute, and monitor repeatable end-to-end ML workflows with an intuitive drag-and-drop UI or the Python SDK. From a SageMaker pipeline, a default MLflow App will be created if one doesn’t already exist, an MLflow experiment name can be defined, and metrics, parameters, and artifacts are logged to the MLflow App as defined in your SageMaker pipeline code. The following screenshot shows an example ML pipeline using MLflow.

SageMaker model customization and MLflow integration
By default, SageMaker model customization integrates with MLflow, providing automatic linking between model customization jobs and MLflow experiments. When you run model customization fine-tuning jobs, the default MLflow App is used, an experiment is selected, and metrics, parameters, and artifacts are logged for you automatically. On the SageMaker model customization job page, you can view metrics sourced from MLflow and drill into additional metrics within the MLflow UI, as shown in the following screenshot.

Conclusion
These features make the new MLflow Apps in SageMaker AI ready for enterprise-scale ML and generative AI workloads with minimal administrative burden. You can get started with the examples provided in the GitHub samples repository and AWS workshop.
MLflow Apps are generally available in the AWS Regions where SageMaker Studio is available, except China and US GovCloud Regions. We invite you to explore the new capability and experience the enhanced efficiency and control it brings to your ML projects. Get started now by visiting the SageMaker AI with MLflow product detail page and Accelerate generative AI development using managed MLflow on Amazon SageMaker AI, and send your feedback to AWS re:Post for SageMaker or through your usual AWS support contacts.

About the authors
Sandeep Raveesh is a GenAI Specialist Solutions Architect at AWS. He works with customers through their AIOps journey across model training, generative AI applications like agents, and scaling generative AI use cases. He also focuses on go-to-market strategies helping AWS build and align products to solve industry challenges in the generative AI space. You can connect with Sandeep on LinkedIn to learn about generative AI solutions.
Rahul Easwar is a Senior Product Manager at AWS, leading managed MLflow and Partner AI Apps within the Amazon SageMaker AIOps team. With over 20 years of experience spanning startups to enterprise technology, he leverages his entrepreneurial background and MBA from Chicago Booth to build scalable ML platforms that simplify AI adoption for organizations worldwide. Connect with Rahul on LinkedIn to learn more about his work in ML platforms and enterprise AI solutions.
Jessica Liao is a Senior UX Designer at AWS who leads design for MLflow, model governance, and inference within Amazon SageMaker AI, shaping how data scientists evaluate, govern, and deploy models. She brings expertise in handling complex problems and driving human-centered innovation from her experience designing DNA life science systems, which she now applies to make machine learning tools more accessible and intuitive through cross-functional collaboration.

The new LiteRT NeuroPilot Accelerator from Google and MediaTek is a concrete step toward running real generative models on phones, laptops, and IoT hardware without shipping every request to a data center. It takes the existing LiteRT runtime and wires it directly into MediaTek’s NeuroPilot NPU stack, so developers can deploy LLMs and embedding models with a single API surface instead of per chip custom code.

What is LiteRT NeuroPilot Accelerator?

LiteRT is the successor of TensorFlow Lite. It is a high performance runtime that sits on device, runs models in .tflite FlatBuffer format, and can target CPU, GPU and now NPU backends through a unified hardware acceleration layer.

LiteRT NeuroPilot Accelerator is the new NPU path for MediaTek hardware. It replaces the older TFLite NeuroPilot delegate with a direct integration to the NeuroPilot compiler and runtime. Instead of treating the NPU as a thin delegate, LiteRT now uses a Compiled Model API that understands Ahead of Time (AOT) compilation and on device compilation, and exposes both through the same C++ and Kotlin APIs.

On the hardware side, the integration currently targets MediaTek Dimensity 7300, 8300, 9000, 9200, 9300 and 9400 SoCs, which together cover a large part of the Android mid range and flagship device space.

Why Developers Care, Unified Workflow For Fragmented NPUs??

Historically, on device ML stacks were CPU and GPU first. NPU SDKs shipped as vendor specific toolchains that required separate compilation flows per SoC, custom delegates, and manual runtime packaging. The result was a combinatorial explosion of binaries and a lot of device specific debugging.

LiteRT NeuroPilot Accelerator replaces that with a three step workflow that is the same regardless of which MediaTek NPU is present:

Convert or load a .tflite model as usual.

Optionally use the LiteRT Python tools to run AOT compilation and produce an AI Pack that is tied to one or more target SoCs.

Ship the AI Pack through Play for On-device AI (PODAI), then select Accelerator.NPU at runtime. LiteRT handles device targeting, runtime loading, and falls back to GPU or CPU if the NPU is not available.

For you as an engineer, the main change is that device targeting logic moves into a structured configuration file and Play delivery, while the app code mostly interacts with CompiledModel and Accelerator.NPU.

AOT and on device compilation are both supported. AOT compiles for a known SoC ahead of time and is recommended for larger models because it removes the cost of compiling on the user device. On device compilation is better for small models and generic .tflite distribution, at the cost of higher first run latency. The blog shows that for a model such as Gemma-3-270M, pure on device compilation can take more than 1 minute, which makes AOT the realistic option for production LLM use.

Gemma, Qwen, And Embedding Models On MediaTek NPU

The stack is built around open weight models rather than a single proprietary NLU path. Google and MediaTek list explicit, production oriented support for:

Qwen3 0.6B, for text generation in markets such as mainland China.

Gemma-3-270M, a compact base model that is easy to fine tune for tasks like sentiment analysis and entity extraction.

Gemma-3-1B, a multilingual text only model for summarization and general reasoning.

Gemma-3n E2B, a multimodal model that handles text, audio and vision for things like real time translation and visual question answering.

EmbeddingGemma 300M, a text embedding model for retrieval augmented generation, semantic search and classification.

On the latest Dimensity 9500, running on a Vivo X300 Pro, the Gemma 3n E2B variant reaches more than 1600 tokens per second in prefill and 28 tokens per second in decode at a 4K context length when executed on the NPU.

For text generation use cases, LiteRT-LM sits on top of LiteRT and exposes a stateful engine with a text in text out API. A typical C++ flow is to create ModelAssets, build an Engine with litert::lm::Backend::NPU, then create a Session and call GenerateContent per conversation. For embedding workloads, EmbeddingGemma uses the lower level LiteRT CompiledModel API in a tensor in tensor out configuration, again with the NPU selected through hardware accelerator options.

Developer Experience, C++ Pipeline And Zero Copy Buffers

LiteRT introduces a new C++ API that replaces the older C entry points and is designed around explicit Environment, Model, CompiledModel and TensorBuffer objects.

For MediaTek NPUs, this API integrates tightly with Android’s AHardwareBuffer and GPU buffers. You can construct input TensorBuffer instances directly from OpenGL or OpenCL buffers with TensorBuffer::CreateFromGlBuffer, which lets image processing code feed NPU inputs without an intermediate copy through CPU memory. This is important for real time camera and video processing where multiple copies per frame quickly saturate memory bandwidth.

A typical high level C++ path on device looks like this, omitting error handling for clarity:

Copy CodeCopiedUse a different Browser// Load model compiled for NPU
auto model = Model::CreateFromFile(“model.tflite”);
auto options = Options::Create();
options->SetHardwareAccelerators(kLiteRtHwAcceleratorNpu);

// Create compiled model
auto compiled = CompiledModel::Create(*env, *model, *options);

// Allocate buffers and run
auto input_buffers = compiled->CreateInputBuffers();
auto output_buffers = compiled->CreateOutputBuffers();
input_buffers[0].Write<float>(input_span);
compiled->Run(input_buffers, output_buffers);
output_buffers[0].Read(output_span);

The same Compiled Model API is used whether you are targeting CPU, GPU or the MediaTek NPU, which reduces the amount of conditional logic in application code.

Key Takeaways

LiteRT NeuroPilot Accelerator is the new, first class NPU integration between LiteRT and MediaTek NeuroPilot, replacing the old TFLite delegate and exposing a unified Compiled Model API with AOT and on device compilation on supported Dimensity SoCs.

The stack targets concrete open weight models, including Qwen3-0.6B, Gemma-3-270M, Gemma-3-1B, Gemma-3n-E2B and EmbeddingGemma-300M, and runs them through LiteRT and LiteRT LM on MediaTek NPUs with a single accelerator abstraction.

AOT compilation is strongly recommended for LLMs, for example Gemma-3-270M can take more than 1 minute to compile on device, so production deployments should compile once in the pipeline and ship AI Packs via Play for On device AI.

On a Dimensity 9500 class NPU, Gemma-3n-E2B can reach more than 1600 tokens per second in prefill and 28 tokens per second in decode at 4K context, with measured throughput up to 12 times CPU and 10 times GPU for LLM workloads.

For developers, the C++ and Kotlin LiteRT APIs provide a common path to select Accelerator.NPU, manage compiled models and use zero copy tensor buffers, so CPU, GPU and MediaTek NPU targets can share one code path and one deployment workflow.

Check out the Docs and Technical details. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post Google LiteRT NeuroPilot Stack Turns MediaTek Dimensity NPUs into First Class Targets for on Device LLMs appeared first on MarkTechPost.

In this tutorial, we explore how an intelligent agent can gradually form procedural memory by learning reusable skills directly from its interactions with an environment. We design a minimal yet powerful framework in which skills behave like neural modules: they store action sequences, carry contextual embeddings, and are retrieved by similarity when a new situation resembles an experience. As we run our agent through multiple episodes, we observe how its behaviour becomes more efficient, moving from primitive exploration to leveraging a library of skills that it has learned on its own. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserimport numpy as np
import matplotlib.pyplot as plt
from collections import defaultdict

class Skill:
def __init__(self, name, preconditions, action_sequence, embedding, success_count=0):
self.name = name
self.preconditions = preconditions
self.action_sequence = action_sequence
self.embedding = embedding
self.success_count = success_count
self.times_used = 0

def is_applicable(self, state):
for key, value in self.preconditions.items():
if state.get(key) != value:
return False
return True

def __repr__(self):
return f”Skill({self.name}, used={self.times_used}, success={self.success_count})”

class SkillLibrary:
def __init__(self, embedding_dim=8):
self.skills = []
self.embedding_dim = embedding_dim
self.skill_stats = defaultdict(lambda: {“attempts”: 0, “successes”: 0})

def add_skill(self, skill):
for existing_skill in self.skills:
if self._similarity(skill.embedding, existing_skill.embedding) > 0.9:
existing_skill.success_count += 1
return existing_skill
self.skills.append(skill)
return skill

def retrieve_skills(self, state, query_embedding=None, top_k=3):
applicable = [s for s in self.skills if s.is_applicable(state)]
if query_embedding is not None and applicable:
similarities = [self._similarity(query_embedding, s.embedding) for s in applicable]
sorted_skills = [s for _, s in sorted(zip(similarities, applicable), reverse=True)]
return sorted_skills[:top_k]
return sorted(applicable, key=lambda s: s.success_count / max(s.times_used, 1), reverse=True)[:top_k]

def _similarity(self, emb1, emb2):
return np.dot(emb1, emb2) / (np.linalg.norm(emb1) * np.linalg.norm(emb2) + 1e-8)

def get_stats(self):
return {
“total_skills”: len(self.skills),
“total_uses”: sum(s.times_used for s in self.skills),
“avg_success_rate”: np.mean([s.success_count / max(s.times_used, 1) for s in self.skills]) if self.skills else 0
}

We define how skills are represented and stored in a memory structure. We implement similarity-based retrieval so that the agent can match a new state with past skills using cosine similarity. As we work through this layer, we see how skill reuse becomes possible once skills acquire metadata, embeddings, and usage statistics. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserclass GridWorld:
def __init__(self, size=5):
self.size = size
self.reset()

def reset(self):
self.agent_pos = [0, 0]
self.goal_pos = [self.size-1, self.size-1]
self.objects = {“key”: [2, 2], “door”: [3, 3], “box”: [1, 3]}
self.inventory = []
self.door_open = False
return self.get_state()

def get_state(self):
return {
“agent_pos”: tuple(self.agent_pos),
“has_key”: “key” in self.inventory,
“door_open”: self.door_open,
“at_goal”: self.agent_pos == self.goal_pos,
“objects”: {k: tuple(v) for k, v in self.objects.items()}
}

def step(self, action):
reward = -0.1
if action == “move_up”:
self.agent_pos[1] = min(self.agent_pos[1] + 1, self.size – 1)
elif action == “move_down”:
self.agent_pos[1] = max(self.agent_pos[1] – 1, 0)
elif action == “move_left”:
self.agent_pos[0] = max(self.agent_pos[0] – 1, 0)
elif action == “move_right”:
self.agent_pos[0] = min(self.agent_pos[0] + 1, self.size – 1)
elif action == “pickup_key”:
if self.agent_pos == self.objects[“key”] and “key” not in self.inventory:
self.inventory.append(“key”)
reward = 1.0
elif action == “open_door”:
if self.agent_pos == self.objects[“door”] and “key” in self.inventory:
self.door_open = True
reward = 2.0
done = self.agent_pos == self.goal_pos and self.door_open
if done:
reward = 10.0
return self.get_state(), reward, done

We construct a simple environment in which the agent learns tasks such as picking up a key, opening a door, and reaching a goal. We use this environment as a playground for our procedural memory system, allowing us to observe how primitive actions evolve into more complex, reusable skills. The environment’s structure helps us observe clear, interpretable improvements in behaviour across episodes. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserclass ProceduralMemoryAgent:
def __init__(self, env, embedding_dim=8):
self.env = env
self.skill_library = SkillLibrary(embedding_dim)
self.embedding_dim = embedding_dim
self.episode_history = []
self.primitive_actions = [“move_up”, “move_down”, “move_left”, “move_right”, “pickup_key”, “open_door”]

def create_embedding(self, state, action_seq):
state_vec = np.zeros(self.embedding_dim)
state_vec[0] = hash(str(state[“agent_pos”])) % 1000 / 1000
state_vec[1] = 1.0 if state.get(“has_key”) else 0.0
state_vec[2] = 1.0 if state.get(“door_open”) else 0.0
for i, action in enumerate(action_seq[:self.embedding_dim-3]):
state_vec[3+i] = hash(action) % 1000 / 1000
return state_vec / (np.linalg.norm(state_vec) + 1e-8)

def extract_skill(self, trajectory):
if len(trajectory) < 2:
return None
start_state = trajectory[0][0]
actions = [a for _, a, _ in trajectory]
preconditions = {“has_key”: start_state.get(“has_key”, False), “door_open”: start_state.get(“door_open”, False)}
end_state = self.env.get_state()
if end_state.get(“has_key”) and not start_state.get(“has_key”):
name = “acquire_key”
elif end_state.get(“door_open”) and not start_state.get(“door_open”):
name = “open_door_sequence”
else:
name = f”navigate_{len(actions)}_steps”
embedding = self.create_embedding(start_state, actions)
return Skill(name, preconditions, actions, embedding, success_count=1)

def execute_skill(self, skill):
skill.times_used += 1
trajectory = []
total_reward = 0
for action in skill.action_sequence:
state = self.env.get_state()
next_state, reward, done = self.env.step(action)
trajectory.append((state, action, reward))
total_reward += reward
if done:
skill.success_count += 1
return trajectory, total_reward, True
return trajectory, total_reward, False

def explore(self, max_steps=20):
trajectory = []
state = self.env.get_state()
for _ in range(max_steps):
action = self._choose_exploration_action(state)
next_state, reward, done = self.env.step(action)
trajectory.append((state, action, reward))
state = next_state
if done:
return trajectory, True
return trajectory, False

We focus on building embeddings that encode the context of a state-action sequence, enabling us to meaningfully compare skills. We also extract skills from successful trajectories, transforming raw experience into reusable behaviours. As we run this code, we observe how simple exploration gradually yields structured knowledge that the agent can apply later. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browser def _choose_exploration_action(self, state):
agent_pos = state[“agent_pos”]
if not state.get(“has_key”):
key_pos = state[“objects”][“key”]
if agent_pos == key_pos:
return “pickup_key”
if agent_pos[0] < key_pos[0]:
return “move_right”
if agent_pos[0] > key_pos[0]:
return “move_left”
if agent_pos[1] < key_pos[1]:
return “move_up”
return “move_down”
if state.get(“has_key”) and not state.get(“door_open”):
door_pos = state[“objects”][“door”]
if agent_pos == door_pos:
return “open_door”
if agent_pos[0] < door_pos[0]:
return “move_right”
if agent_pos[0] > door_pos[0]:
return “move_left”
if agent_pos[1] < door_pos[1]:
return “move_up”
return “move_down”
goal_pos = (4, 4)
if agent_pos[0] < goal_pos[0]:
return “move_right”
if agent_pos[1] < goal_pos[1]:
return “move_up”
return np.random.choice(self.primitive_actions)

def run_episode(self, use_skills=True):
self.env.reset()
total_reward = 0
steps = 0
trajectory = []
while steps < 50:
state = self.env.get_state()
if use_skills and self.skill_library.skills:
query_emb = self.create_embedding(state, [])
skills = self.skill_library.retrieve_skills(state, query_emb, top_k=1)
if skills:
skill_traj, skill_reward, success = self.execute_skill(skills[0])
trajectory.extend(skill_traj)
total_reward += skill_reward
steps += len(skill_traj)
if success:
return trajectory, total_reward, steps, True
continue
action = self._choose_exploration_action(state)
next_state, reward, done = self.env.step(action)
trajectory.append((state, action, reward))
total_reward += reward
steps += 1
if done:
return trajectory, total_reward, steps, True
return trajectory, total_reward, steps, False

def train(self, episodes=10):
stats = {“rewards”: [], “steps”: [], “skills_learned”: [], “skill_uses”: []}
for ep in range(episodes):
trajectory, reward, steps, success = self.run_episode(use_skills=True)
if success and len(trajectory) >= 3:
segment = trajectory[-min(5, len(trajectory)):]
skill = self.extract_skill(segment)
if skill:
self.skill_library.add_skill(skill)
stats[“rewards”].append(reward)
stats[“steps”].append(steps)
stats[“skills_learned”].append(len(self.skill_library.skills))
stats[“skill_uses”].append(self.skill_library.get_stats()[“total_uses”])
print(f”Episode {ep+1}: Reward={reward:.1f}, Steps={steps}, Skills={len(self.skill_library.skills)}, Success={success}”)
return stats

We define how the agent chooses between using known skills and exploring with primitive actions. We train the agent across several episodes and record the evolution of learned skills, usage counts, and success rates. As we examine this part, we observe that skill reuse reduces episode length and improves overall rewards. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserdef visualize_training(stats):
fig, axes = plt.subplots(2, 2, figsize=(12, 8))
axes[0, 0].plot(stats[“rewards”])
axes[0, 0].set_title(“Episode Rewards”)
axes[0, 1].plot(stats[“steps”])
axes[0, 1].set_title(“Steps per Episode”)
axes[1, 0].plot(stats[“skills_learned”])
axes[1, 0].set_title(“Skills in Library”)
axes[1, 1].plot(stats[“skill_uses”])
axes[1, 1].set_title(“Cumulative Skill Uses”)
plt.tight_layout()
plt.savefig(“skill_learning_stats.png”, dpi=150, bbox_inches=’tight’)
plt.show()

if __name__ == “__main__”:
print(“=== Procedural Memory Agent Demo ===n”)
env = GridWorld(size=5)
agent = ProceduralMemoryAgent(env)
print(“Training agent to learn reusable skills…n”)
stats = agent.train(episodes=15)
print(“n=== Learned Skills ===”)
for skill in agent.skill_library.skills:
print(f”{skill.name}: {len(skill.action_sequence)} actions, used {skill.times_used} times, {skill.success_count} successes”)
lib_stats = agent.skill_library.get_stats()
print(f”n=== Library Statistics ===”)
print(f”Total skills: {lib_stats[‘total_skills’]}”)
print(f”Total skill uses: {lib_stats[‘total_uses’]}”)
print(f”Avg success rate: {lib_stats[‘avg_success_rate’]:.2%}”)
visualize_training(stats)
print(“n✓ Skill learning complete! Check the visualization above.”)

We bring everything together by running training, printing learned skills, and plotting behaviour statistics. We visualize the trend in rewards and how the skill library grows over time. By running this snippet, we complete the lifecycle of procedural memory formation and confirm that the agent learns to behave more intelligently with experience.

In conclusion, we see how procedural memory emerges naturally when an agent learns to extract skills from its own successful trajectories. We observe how skills are gained, structure, metadata, embeddings, and usage patterns, allowing the agent to reuse them efficiently in future situations. Lastly, we appreciate how even a small environment and simple heuristics lead to meaningful learning dynamics, giving us a concrete understanding of what it means for an agent to develop reusable internal competencies over time.

Check out the FULL CODES here. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post A Coding Guide to Build a Procedural Memory Agent That Learns, Stores, Retrieves, and Reuses Skills as Neural Modules Over Time appeared first on MarkTechPost.

Artificial intelligence (AI) reasoning capabilities determine whether models can handle complex, real-world tasks beyond simple pattern matching. With strong reasoning, models can identify problems from ambiguous descriptions, apply policies under competing constraints, adapt tone to sensitive situations, and provide complete solutions that address root causes. Without robust reasoning, AI systems fail when faced with nuanced scenarios requiring judgment, context awareness, and multi-step problem-solving.
This post evaluates the reasoning capabilities of our latest offering in the Nova family, Amazon Nova Lite 2.0, using practical scenarios that test these critical dimensions. We compare its performance against other models in the Nova family—Lite 1.0, Micro, Pro 1.0, and Premier—to elucidate how the latest version advances reasoning quality and consistency.
Solution overview
We evaluate five Amazon Nova models across five customer support scenarios, measuring performance on eight dimensions:

Problem identification
Solution completeness
Policy adherence
Factual accuracy
Empathy and tone
Communication clarity
Logical coherence
Practical utility

An independent evaluator model (gpt-oss-20b) provides automated, unbiased scoring.
The evaluation architecture uses the same Region: us-east-1 and automatically handles different API formats: Converse API for Nova, OpenAI Chat Completions for gpt-oss-20b.
The sample notebook is available in the GitHub repository.
Test scenarios
To generate the scenarios evaluation dataset, we use Claude Sonnet 4.5 by Anthropic on Amazon Bedrock to generate a sample of 100 scenarios that pertain to common customer support interactions. We don’t use any of the Nova models to generate the scenarios to avoid any bias. We then randomly select five scenarios for our testing purposes that evaluate common real-world reasoning challenges:

Angry customer complaint – Tests de-escalation, empathy, and problem resolution when a customer threatens to leave after delayed delivery and poor service.
Software technical problem – Evaluates technical troubleshooting when an app crashes during photo uploads despite basic troubleshooting attempts.
Billing dispute – Assesses investigation skills and security awareness for unrecognized charges potentially indicating unauthorized access.
Product defect report – Measures warranty policy application and customer service for a two-month-old defective product.
Account security concern – Tests urgency response and security protocols for unauthorized password changes and fraudulent purchases.

Each scenario includes key issues to identify, required solutions, and relevant policies—providing objective criteria for evaluation. Depending on your industry/domain/use case, the scenarios and associated context may be different.
Implementation details
The evaluation framework establishes a comprehensive methodology for assessing model performance across multiple dimensions simultaneously. This systematic approach ensures that each model undergoes identical testing conditions, enabling fair comparison of reasoning capabilities across the Nova family. The technical implementation handles the complexity of managing different API formats while maintaining evaluation consistency. The framework assumes an active AWS account, access to Nova models and gpt-oss-20b, along with the availability of the boto3 SDK, and pandas, matplotlib, seaborn, scipy and numpy packages.
Model invocation
The system automatically detects which API format each model requires and routes requests accordingly. Nova models (Lite, Micro, Pro, Premier) use Amazon Bedrock Converse API, which provides a unified interface for conversational interactions. gpt-oss models use the OpenAI Chat Completions format, requiring a different request structure with the InvokeModel API. The invocation function checks the model identifier to determine the appropriate format. For gpt-oss models, it constructs a JSON request body with messages, token limits, and temperature settings, then parses the response to extract the generated content. For Nova models, it uses the Converse API with structured message objects and inference configuration parameters, extracting the response from the output message content. This dual-API approach supports seamless evaluation across different model families without requiring separate code paths or manual configuration changes. The same evaluation logic works for all models regardless of their underlying API requirements, with the system handling format differences transparently. The architecture also allows us to use models from different Regions while maintaining a single evaluation workflow.
The evaluation framework uses optimized prompts generated by the Amazon Bedrock Prompt Optimizer API. The optimizer analyzes and rewrites raw prompts to improve model performance with better structure, clarity, and organization, creating model-specific optimizations for each Nova model.
A scenario with the optimized prompt is shown in the following example:

“`json
{
“angry_customer”: {
“name”: “Angry Customer Complaint”,
“prompt”: “# Customer Support Response Tasknn## ContextnYou are a professional customer support representative for a technology company. You need to respond to an upset customer who has written the following message:nn”I am absolutely furious! I ordered a laptop 3 weeks ago and it still hasn’t arrived. When I called last week, your representative was rude and unhelpful. I’ve been a loyal customer for 5 years and this is how you treat me? I want my money back immediately and I’m considering switching to your competitor. This is unacceptable!”nn## InstructionsnCraft a professional, empathetic response that:n1. Acknowledges the customer’s frustration and validates their feelingsn2. Apologizes sincerely for the specific issues (delayed delivery and poor customer service)n3. Demonstrates understanding of their value as a loyal 5-year customern4. Offers a clear solution to address their refund requestn5. Provides a specific action plan to resolve the delivery issue (if they choose not to cancel)n6. Includes a concrete step to follow up and rebuild trustn7. Maintains a respectful, professional tone throughoutnnYour response should be concise, solution-oriented, and focused on retaining this valuable customer. Avoid making excuses or shifting blame.nnProvide your response immediately without any preamble.”,
“key_issues”: [
“Delayed delivery”,
“Poor customer service experience”,
“Customer loyalty concerns”,
“Refund request”
],
“required_solutions”: [
“Apologize sincerely”,
“Investigate delivery status”,
“Offer compensation”,
“Escalate if needed”
],
“policies”: [
“Always acknowledge customer emotions”,
“Provide specific next steps”,
“Offer multiple resolution options”
],
“_optimization_metadata”: {
“original_length”: 463,
“optimized_length”: 1330,
“target_model”: “amazon.nova-2-lite-v1:0”
}
}
}
“`

Evaluation Framework
The evaluator receives the scenario, model response, and evaluation criteria. We employ a two-step scoring process: first, the evaluator assigns a category label that best characterizes the response; then, the evaluator assigns a predetermined score corresponding to that category label. This approach ensures a consistent and uniform scoring methodology across all model responses.
The evaluation prompt structure:

“`python
EVALUATION_PROMPT = “””
# Customer Support Response Evaluation Task

You are an expert evaluator assessing customer support responses. Your task is to
provide **detailed, objective scoring** across 8 dimensions with specific reasoning
for each score.

—

## Context

### Original Customer Scenario
{scenario}

### Model’s Response to Evaluate
{response}

—

## Evaluation Criteria

### Key Issues That Should Be Identified
{key_issues}

### Required Solutions/Actions
{required_solutions}

### Company Policies to Follow
{policies}

—

## Scoring Instructions

Evaluate the response across **8 dimensions** using a **two-step process**:

### Step 1: Assign Category Label

For each dimension, first determine which category best describes the response:

**EXCELLENT**: Comprehensive, professional, exceeds expectations
– All requirements fully met with exceptional quality
– No significant improvements needed
– Demonstrates mastery of the dimension

**GOOD**: Solid performance with minor room for improvement
– Most requirements met effectively
– Minor gaps or areas for enhancement
– Clearly competent but not exceptional

**ADEQUATE**: Meets basic requirements but has notable gaps
– Core requirements partially met
– Significant room for improvement
– Functional but not impressive

**POOR**: Significant issues requiring major improvements
– Many requirements not met
– Critical gaps in quality
– Barely functional or ineffective

**FAILING**: Critical failures, does not meet requirements
– Fundamental requirements not met
– Unusable or harmful response
– Complete failure on this dimension

### Step 2: Assign Fixed Score

Each category maps to a fixed score:
– **EXCELLENT** → 10
– **GOOD** → 8
– **ADEQUATE** → 6
– **POOR** → 4
– **FAILING** → 2

For **EACH dimension**, provide:
1. **Category label** (EXCELLENT/GOOD/ADEQUATE/POOR/FAILING)
2. **Fixed score** (10/8/6/4/2 based on category)
3. **Specific reasoning** explaining your categorization

—

## Evaluation Dimensions

### 1. Problem Identification
**Question**: Did the response identify all key issues from the customer’s message?
– Check if all items from “Key Issues” were recognized
– Note any missed or misunderstood problems

### 2. Solution Completeness
**Question**: Are all identified problems addressed with appropriate solutions?
– Verify each issue has a corresponding solution or action
– Check if solutions are practical and actionable

### 3. Policy Adherence
**Question**: Does the response follow all stated company policies?
– Review against “Company Policies to Follow”
– Note any policy violations or omissions

### 4. Factual Accuracy
**Question**: Are technical details, processes, and options stated correctly?
– Check for factual errors or misleading information
– Verify technical accuracy of troubleshooting steps

### 5. Empathy & Tone
**Question**: Does the response demonstrate appropriate emotional intelligence?
– Assess acknowledgment of customer emotions
– Evaluate professionalism and empathy level

### 6. Communication Clarity
**Question**: Is the response clear, well-structured, and actionable?
– Check for clear language and organization
– Verify instructions are easy to follow

### 7. Logical Coherence
**Question**: Is the reasoning sound without contradictions?
– Look for logical flow and consistency
– Identify any contradictory statements

### 8. Practical Utility
**Question**: Would this response actually help the customer resolve their issue?
– Consider real-world effectiveness
– Assess likelihood of customer satisfaction

—

## Example Evaluation
<>
“””
“`

The evaluator must justify scores, providing transparency into the assessment. To address transparency concerns in AI evaluation, the evaluator provides detailed reasoning for each of the eight dimensions, plus an overall justification. This ensures that scores are not just numerical but backed by specific explanations of why each score was assigned.
Large language model (LLM)-as-a-judge evaluation
Machine translation-based evaluation techniques like ROUGE and BLEU fall short when it comes to open ended conversations. LLM-as-a-judge provides scalability, flexibility and evaluations that closely match human preferences up to 80%.
Refer to the comparison table in the README for further details.
Evaluation process
For each model and scenario combination, we perform 10 runs to measure consistency. This produces 250 evaluations (5 models × 5 scenarios × 10 runs) providing a statistical spread through multiple measurements. The number of runs and scenarios can be increased according to the specific use case. The framework includes diagnostic checks to verify evaluation quality and reliability. Failed evaluations (where the evaluator returns a score of 0 due to technical issues such as JSON parsing errors, or when models don’t respond owing to blocked responses adhering to Responsible AI criteria) are excluded from mean and standard deviation calculations to ensure accurate performance metrics. This prevents technical failures from artificially lowering model scores.
Results
The chosen scenarios and approach described here enable deep statistical analysis of model performance patterns. By examining both individual scenario outcomes and aggregate metrics, we can identify strengths and potential areas for improvement across the Nova model family. This multi-dimensional analysis approach provides confidence in the reliability of performance rankings.
Statistical analysis
The statistical evaluation we use follow the methods outlined in Miller, 2024. To quantify uncertainty in model performance estimates, we calculate standard error (SE) as:

SE = √(σ^2/n),

where σ^2 is the sample variance, and n is the sample size. SE measures how precise our estimate of the mean is and tells us how much the sample mean would vary if we repeated the evaluation many times. The standard error allows us to construct 95% confidence intervals (CI = μ± 1.96×SE), where μ is the sample mean. This provides plausible ranges for true model performance, facilitating statistical significance testing through interval overlap analysis. In addition, we introduce a coefficient of variation (CV) based consistency score calculated as (100 – CV%), where CV% = (σ/μ)×100, and σ is the standard deviation. This normalizes reliability measurement on a 0-100 scale, thereby providing an intuitive metric for response stability. Finally, zero-exclusion averaging prevents failed evaluations from artificially deflating scores, while error bars on visualizations transparently communicate uncertainty. For the sake of completeness, the code in the GitHub repository calculates other statistics such as a minimum detectable effect that demonstrates the ability to reliably detect meaningful performance differences, a pairwise model comparison metric that identifies correlations between model responses, and a power analysis that validates the chosen sample size. These methodologies transform the evaluation from simple score comparison into rigorous experimental science with quantified uncertainty, enabling confident conclusions about model performance differences.

Figure 1 Performance of models across the dimensions considered in the study with 95% confidence intervals

Figure 2 Overall performance of Nova Lite 2.0 compared to other models in the Nova family
Figure 1 shows the performance of models with scores averaged across all the runs for each dimension considered in the study; this is also depicted on the radar chart in Figure 2. Table 1 shows the scores across all dimensions considered in the study. Nova Lite 2.0 achieved the highest overall score (9.42/10) with a standard error of 0.08 and a coefficient of variation of 5.55%, demonstrating high-quality reasoning.

Metric
Nova Lite 2.0
Nova Lite 1.0
Nova Pro 1.0
Nova Micro
Nova Premier

Overall Score
9.42
8.65
8.53
7.70
7.16

Standard Error (SE)
0.08
0.09
0.12
0.32
0.38

95% Confidence Interval
[9.28, 9.57]
[8.48, 8.82]
[8.30, 8.76]
[7.08, 8.32]
[6.41, 7.91]

Consistency Score (CV-based)
94.45
93.05
90.46
71.37
62.96

Coefficient of Variation
5.55%
6.95%
9.54%
28.63%
37.04%

Table 1: Overall Model Performance Summary

Metric
Nova Lite 2.0
Nova Lite 1.0
Nova Pro 1.0
Nova Micro
Nova Premier

Problem Identification
9.63 ± 0.27
8.57 ± 0.46
8.16 ± 0.44
7.59 ± 0.74
6.94 ± 0.82

Solution Completeness
9.59 ± 0.23
8.08 ± 0.32
8.04 ± 0.42
6.78 ± 0.65
6.33 ± 0.69

Policy Adherence
8.82 ± 0.54
7.76 ± 0.59
7.55 ± 0.64
7.02 ± 0.69
6.37 ± 0.81

Factual Accuracy
9.55 ± 0.26
9.18 ± 0.30
9.10 ± 0.28
8.08 ± 0.74
8.00 ± 0.89

Empathy Tone
8.98 ± 0.33
8.57 ± 0.34
8.08 ± 0.36
7.55 ± 0.65
7.10 ± 0.79

Communication Clarity
9.76 ± 0.19
9.14 ± 0.28
8.94 ± 0.28
8.04 ± 0.69
7.63 ± 0.85

Logical Coherence
9.71 ± 0.35
9.67 ± 0.29
9.92 ± 0.11
8.98 ± 0.74
8.16 ± 0.91

Practical Utility
9.35 ± 0.27
8.24 ± 0.22
8.45 ± 0.24
7.55 ± 0.62
6.78 ± 0.70

Table 2: Dimension-Level Performance of the Nova models (Mean Scores with 95% Confidence Intervals)
Table 2 shows the performance across the eight dimensions considered in the study. Nova Lite 2.0 achieved consistently high scores across all dimensions.

Scenario
Nova Lite 2.0
Nova Lite 1.0
Nova Micro
Nova Pro 1.0
Nova Premier

Account Security Concern
9.25
7.95
7.65
6.90
2.00

Angry Customer Complaint
9.95
9.50
9.30
8.35
8.20

Billing Dispute
9.15
8.75
8.60
8.85
8.20

Product Defect Report
9.25
8.90
7.70
8.00
8.75

Software Technical Problem
10.00
8.20
8.55
8.75
8.60

Table 3 Summary of scores (on a scale of 1-10) across models and scenarios considered. A score of 2 for Nova Premier for Account Security Concern is due to Guardrails being invoked for almost all of the responses.
Table 3 summarizes the mean scores corresponding to each scenario considered in the study. Again, Nova Lite 2.0 achieves high scores across all dimensions.
Dimension analysis
The dimensional strengths of Nova Lite 2.0 demonstrate balanced capabilities across critical evaluation criteria. High scores in problem identification, communication, and logical reasoning indicate mature performance that translates effectively to real-world applications, distinguishing it from models that excel in individual dimensions but lack consistency.
Problem Identification: Nova Lite 2.0 excelled at identifying all key issues—crucial where missing problems lead to incomplete solutions.
Communication Clarity: The model achieved the highest score in this dimension, producing well-structured, actionable responses customers could follow easily.
Logical Coherence: Strong performance indicates the model maintains sound reasoning without contradictions across complex scenarios.
Empathy and Tone: High scores demonstrate appropriate emotional intelligence, critical for de-escalation and sensitive situations.
Table 4 shows sample evaluator explanations for high-scoring and low-scoring models, illustrating effective scoring methodology.

Nova Lite 2.0 – Score: 10 – Category: “Excellent” The response explicitly recognizes the four key issues: it mentions the delayed delivery (“delay in receiving your laptop”), the poor customer service experience (“unhelpful interaction with our support team”), the customer’s loyalty (“a valued customer of five years”), and the refund request (“cancel your order and receive a full refund”). All issues are acknowledged with appropriate language. Nova Premier – Score: 6 – Category: “Adequate” The response acknowledges frustration and loyalty, but it does not explicitly mention the delayed delivery or the rude customer‚ service representative, two key issues from the customer message.

Table 4 Sample explanations provided by the evaluator for Nova Lite 2.0 and Nova Premier for the Angry Customer scenario along the Problem Identification dimension
Key findings
The evaluation results reveal critical insights for model selection and deployment strategies. These findings emphasize considering multiple performance factors rather than focusing solely on aggregate scores, as optimal choices depend on specific application requirements and operational constraints.

Multi-dimensional reasoning matters: Models scoring well on accuracy but poorly on empathy or clarity are unsuitable for customer-facing applications. The balanced performance of Nova Lite 2 across all dimensions makes it production-ready.
Consistency predicts production success: The low variability of Nova Lite 2.0 versus other models indicates reliable performance across diverse scenarios—critical where inconsistent responses damage user trust.
Real-world evaluation reveals practical capabilities: Synthetic benchmarks miss critical dimensions like empathy, policy adherence, and practical utility. This framework surfaces production-relevant capabilities.

Implementation considerations
Successfully implementing this evaluation framework requires attention to operational factors that significantly impact assessment quality and cost-effectiveness. The choice of evaluation methodology, scoring mechanisms, and technical infrastructure directly influences result reliability and scalability.

Evaluator selection: We selected gpt-oss-20b to ensure independence from the Nova family, reducing potential bias. Amazon Bedrock offers built-in LLM-as-a-judge capabilities with standard metrics like correctness, completeness, and harmfulness. The framework presented in this post provides the flexibility to define specialized evaluation criteria and multi-dimensional assessments that can be customized to the specific use case of interest.
Scenario design: Effective scenarios balance realism with measurability. Each includes specific details grounding evaluation in realistic contexts. Objective criteria—key issues to identify, required solutions, relevant policies—enable consistent scoring. Realistic complexity combining multiple problems (billing dispute + security breach) and competing priorities (urgency vs protocols) reveals how models handle real-world ambiguity and surfaces capability gaps.
Statistical validation: Multiple runs per scenario provide confidence intervals and detect inconsistency, ensuring performance differences are statistically significant.

Key takeaways
Amazon Nova Lite 2.0 demonstrates impressive reasoning capabilities in tested real-world scenarios, achieving consistent high performance across diverse problem-solving tasks. Balanced scores across evaluation dimensions—from technical problem identification to empathetic communication—indicate robust reasoning potentially applicable to other domains after comprehensive testing. Multi-dimensional evaluation reveals nuanced model capabilities that single-metric benchmarks miss. Understanding performance across problem identification, solution completeness, policy adherence, empathy, clarity, and logical coherence provides actionable deployment insights. This practical testing methodology provides actionable insights for organizations evaluating AI systems. The framework’s focus on objective criteria, independent evaluation, and statistical validation creates reproducible assessments adaptable to domains requiring contextual judgment and problem-solving. As models advance, assessment methodologies must evolve to capture increasingly sophisticated reasoning capabilities—multi-turn conversations, complex decision-making under uncertainty, and nuanced judgment in ambiguous situations.
Conclusion
This comprehensive evaluation demonstrates that Amazon Nova Lite 2.0 delivers production-ready AI reasoning capabilities with measurable reliability across diverse business applications. The multi-dimensional assessment framework provides organizations with quantitative evidence needed to confidently deploy AI systems in critical operational environments.
Next steps
Evaluate Nova Lite 2.0 for your use case:

Bedrock Model Evaluation: Start with model evaluation tools of Amazon Bedrock, including the built-in LLM-as-a-judge capabilities for standard metrics, or adapt the custom framework discussed in this post for specialized evaluation criteria.
Implement multi-dimensional testing: Adapt the evaluation framework to your specific domain requirements.
Pilot deployment: Begin with low-risk scenarios to validate performance in your environment.
Scale systematically: Use the statistical validation approach to expand to additional use cases.

Additional resources

Miller, E; Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations
Amazon Bedrock Documentation
Amazon Nova models
GitHub repository
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

About the authors
Madhu Pai, Ph.D., is a Principal Specialist Solutions Architect for Generative AI and Machine Learning at AWS. He leads strategic AI/ML initiatives that deliver scalable impact across diverse industries by identifying customer needs and building impactful solutions. Previously at AWS, Madhu served as the WW Partner Tech Lead for Manufacturing where he delivered compelling partner solutions that drove strategic outcomes for industrial manufacturing customers. He brings over 18 years of experience across multiple industries, leveraging data, AI, and ML to deliver measurable business results.
Sunita Koppar is a Senior Specialist Solutions Architect in Generative AI and Machine Learning at AWS, where she partners with customers across diverse industries to design solutions, build proof-of-concepts, and drive measurable business outcomes. Beyond her professional role, she is deeply passionate about learning and teaching Sanskrit, actively engaging with student communities to help them upskill and grow.
Satyanarayana Adimula is a Senior Builder in the AWS GenAI Invocation Center. With over 20 years of experience in data and analytics and deep expertise in generative AI, he helps organizations achieve measurable business outcomes. He builds agentic AI systems that automate workflows, accelerate decision-making, reduce costs, increase productivity, and create new revenue opportunities. His work spans large enterprise customers across various industries, including retail, banking, financial services, insurance, healthcare, media and entertainment, and professional services.

Teams need instant access to enterprise data and intelligent guidance on how to use it. Instead, they get scattered information across multiple systems. This results in employees spending valuable time searching for answers instead of making decisions.
In this post, we show how to build chat agents in Amazon Quick Suite to address this problem. We walk through a three-layer framework—identity, instructions, and knowledge—that transforms Quick Suite chat agents into intelligent enterprise AI assistants. In our example, we demonstrate how our chat agent guides feature discovery, use enterprise data to inform recommendations, and tailors solutions based on potential to impact and your team’s adoption readiness.
Benefits of Quick Suite chat agents
Quick Suite chat agents make advanced AI capabilities accessible to non-technical business users. Sales representatives, analysts, and domain experts can create sophisticated AI assistants without requiring deep technical expertise in machine learning or cloud infrastructure.
Quick Suite instances come with their own default system chat agent (My Assistant). Administrators can enable the ability to create custom chat agents for the users. Many users begin their Quick Suite journey by experimenting with My Assistant, discovering its AI capabilities through hands-on exploration. Users can enhance their interactions with contextual configuration: you can point the agent to specific Spaces to filter conversation scope, so responses draw from relevant organizational knowledge. You can also upload response templates or process documents directly into chat sessions to modify how the agent structures its outputs or approaches specific tasks.
Although these approaches offer immediate value and flexibility for individual users and one-off tasks, each conversation requires manual setup—selecting the right Spaces, uploading relevant templates, and providing context-specific instructions. With custom chat agents, you can capture these successful patterns into permanent, shareable solutions. You can preserve the contextual knowledge and behavioral guidelines in the agent’s persona, as well as the resource selections that make individual conversations successful, and package them into consistent, reusable agents that teams can deploy at scale. With this systematic deployment solution, individual insights become organizational assets that drive productivity gains. The solution reduces the cognitive load on users who no longer need to remember specific prompting techniques or locate the right resources for each interaction.
The three-layer foundation: Identity, instructions, and knowledge
Effective chat agents are built on three essential components that work together to create consistent, reliable AI assistants:

Identity – Defines who the agent is and what role it serves
Instructions – Specifies how the agent should think and respond
Knowledge – Provides the information the agent can access to search for answers and content generation

Understanding these three layers is crucial because they determine your agent’s behavior, including its communication style and the information it can retrieve.
Identity
Identity defines who your agent is and what role it plays, which shapes how it responds to every request. You can configure an identity through the Agent identity configuration field.
Instructions
Instructions function as behavioral directives that provide granular control over agent response generation, with specificity and consistency being crucial for effectiveness. Effective prompt engineering skills become essential when crafting both identity and instructions, because the precision and clarity of these elements directly impact the agent’s ability to understand context, follow behavioral directives, and maintain consistent, persona-driven responses. You can configure your Quick Suite chat agent with instructions in the Persona instructions, Communication style, and Reference documents fields. Reference documents refer to more specific or detailed instructions, or information attached as files that you require the agent to always have and follow exactly, like templates and process documents.
Knowledge
Large language models (LLMs) power the agents. The custom chat agent provides required context to LLMs through two distinct means: instructions as discussed in previous section, and searchable knowledge. Quick Spaces provides the ability to pool searchable knowledge for the chat agent in different forms:

Direct file uploads (indexed knowledge)
Amazon Quick Sight dashboards and topics
Knowledge bases created from data access integrations (indexed knowledge)
Action connectors to take actions on integrated third-party tools

Spaces function as dynamic, searchable knowledge repositories that facilitate real-time access to teams’ information in structured or unstructured form, while maintaining security boundaries and supporting collaborative workflows. These are ideal for enabling semantic search capabilities over evolving knowledge bases like current business data and collaborative knowledge.
Solution overview
The Quick Suite Product Specialist is a custom chat agent to help users identify the right Quick Suite features for their specific needs. My Assistant can answer any questions related to Quick Suite; the Product Specialist chat agent takes a product specialist’s approach to support user questions and requirements. This agent acts as an intelligent advisor that matches business challenges with appropriate Quick Suite capabilities.
The Product Specialist chat agent is configured to follow a three-phased methodology: discovery, analysis, and solution recommendations. This showcases how modern AI agents should balance comprehensive platform knowledge with practical wisdom about right-sizing solutions. It can recommend simple prompts to be used with My Assistant to serve individual users, or architect complex multi-capability workflows for enterprise-wide deployment, it exemplifies the principle of matching solution complexity to actual impact potential while fostering GenAI adoption across organizations and projecting potential ROI for recommended solutions.
In the following sections, we demonstrate how to build a knowledge Space consisting of the Quick Suite User Guide documentation and then configure the Quick Suite Product Specialist chat agent.
Prerequisites
To build a custom chat agent in Quick Suite, you must have the following:

An active Quick Suite instance
A Quick Suite subscription for the required capabilities:

Professional – Create, configure, and share, spaces and custom chat agents
Enterprise (includes Professional capabilities) – Create knowledge bases

For more information about Quick Suite’s subscription tiers, see Amazon Quick Suite pricing.
Create Space with knowledge base
We first set up a Quick Space as part of the context component of the three-layered foundation we discussed previously. This Space contains a searchable knowledge base for the Amazon Quick Suite User Guide.
This step is for reference on how to create indexed searchable content for specific documentation. Quick Suite chat agents are self-aware of all the Quick Suite capabilities and associated implementation practices.
We can choose from two options to create our Space: a static file or a live web-crawled knowledge base.
Use a static file
This option is a static snapshot of the official Quick Suite User Guide and must be updated occasionally to incorporate latest changes and additions to the platform documentation. Complete the following steps:

Go to Amazon Quick Suite User Guide.
Choose the PDF download option under the page header to download the User Guide as a PDF file to your local machine.

On the Quick Suite console, choose Spaces in the navigation pane.
Choose Create space to create a new Space:

For Title, enter a title, such as the following:

Amazon Quick Suite Documentation Space

For Description, enter a description, such as the following:

This Quick Space contains Amazon Quick Suite User Guide file.

Choose Add knowledge and choose File uploads.
Upload the User Guide PDF.
Choose Share to manage Viewer/Owner access to the created Space.

Files uploaded to a Space use the same access permissions as the Space.

Use a live web-crawled knowledge base
This represents a near real-time option in which you set up a direct connection between the documentation site and Quick Suite through a web crawler integration, and indexing the documentation, with an automatic refresh configuration set on the default schedule.

On the Quick Suite console, choose Integrations in the navigation pane.
Choose Add and choose Webcrawler to add a webcrawler.

For Name, use the default name.
Select No authentication.
Choose Create and continue.

Configure the knowledge base:

For Name, enter a name, such as the following:

Amazon Quick Suite User Guide Documentation KB

For Add URLs, enter the main documentation URL:

https://docs.aws.amazon.com/quicksuite/latest/userguide/

Choose Add.
Choose Create.
On the Knowledge bases tab, choose the knowledge base you created. The knowledge base refresh is initiated automatically.
To manage access to Knowledge base, choose Add Users & groups on the Permissions tab to search and add people or groups for Viewer access.

Choose Spaces in the navigation pane.
Choose Create space to create a new Space:

For Title, enter a title, such as the following:

Amazon Quick Suite Documentation Space

For Description, enter a description, such as the following:

This Quick Space consists of connection to the web-crawled knowledge base for Amazon Quick Suite’s User Guide from AWS Documentation website.

Choose Add knowledge, then choose Knowledge bases.
Locate the knowledge base you created and choose Add.
Choose Share to manage Viewer/Owner access to the created Space.

Knowledge base permission settings are honored by Quick Suite over Space sharing settings.
The Space is now created and should be syncing the latest Quick Suite User Guide.

Create chat agent
Complete the following steps to build your own Quick Suite Product Specialist:

On the Quick Suite console, choose Chat agents in the navigation pane.
Choose Create chat agent
Choose Skip to enter Builder view to create a custom chat agent, because we know exactly what instructions and assets the chat agent needs.

For Title, enter a title, such as the following:

Quick Suite Product Specialist

For Description, enter a description, such as the following:

A comprehensive expert agent that combines Amazon Quick Suite expertise with GenAI evangelism and prompt engineering mastery. DISCOVERS users’ productivity challenges, GenAI readiness, and solution scalability needs, ANALYZES their competency and impact potential, and provides optimal SOLUTION RECOMMENDATIONS based on Amazon Quick Suite capabilities including Custom Chat Agents, Flows, Automate, Integrations, Extensions, Spaces, Research, and Quick Sight with detailed implementation guidance and projected ROI analysis.

Update the AGENT PERSONA configuration:

For Agent identity, enter details such as the following:

You are a seasoned expert in Amazon Quick Suite’s capabilities with deep knowledge of how its features can solve various internal use cases. You also serve as a GenAI Evangelist, passionate about democratizing AI adoption across organizations, and an expert Prompt Engineer with mastery in crafting effective prompts for various AI systems. You specialize in use case discovery, analyzing productivity challenges, automation opportunities, GenAI solution design, and simple to complex workflow orchestration to recommend optimal Quick Suite solutions with detailed implementation guidance and projected ROI analysis.
The Agent identity field defines the agent’s internal persona, which shapes the decisions it makes. Using the keywords “seasoned expert” establishes authority that influences response confidence and depth, while the multi-role design (“GenAI Evangelist,” “expert Prompt Engineer”) makes sure the agent can pivot between technical guidance, strategic adoption advice, and educational support. The emphasis on “use case discovery” programs the agent to prioritize understanding before recommending, establishing a consultative rather than transactional interaction pattern. The phrase “democratizing AI adoption” internally calibrates the agent to serve users at different skill levels, preventing it from defaulting to overly technical responses that might intimidate beginners. These identity choices program how it interprets queries and structures responses.
For Persona instructions, enter instructions such as the following:

For each user problem follow this 3-phased approach:
A. DISCOVERY
1. Analyze the initial use case details provided
2. Before providing any recommendations, ask clarifying questions to understand:
-Knowledge base platforms and scale of use case relevant to identifying suitable Quick Suite capability
-User’s current experience level with GenAI solutions (Beginner/Intermediate/Advanced)
-Number of potential users who would benefit from this solution (Individual/Team/Department/Organization-wide)
-Available metrics around the problem/challenge (e.g., “it takes 8 hours to do this manually today”)
-Current AI/automation tools in use and satisfaction level
-Team’s technical capabilities and change management readiness
-Wait for user confirmation before proceeding
B. ANALYSIS
1. Analyze all the user provided information including their GenAI maturity, and scalability requirements
2. Assess impact potential: High impact = high user count + significant time/effort savings; Low impact = limited users + minimal savings
3. Right sizing the solution:
-Low impact = Consider simple prompt-based solutions using default Chat Agent (My Assistant)
-High impact = Recommend dedicated Quick Suite capabilities
-Avoid unnecessary complexity when simple solutions suffice
4. Calculate potential ROI in terms of as time savings by user count
5. CAPABILITY VERIFICATION PROTOCOL:
– Before recommending any specific Quick Suite feature, verify the exact capability exists in available documentation
– Clearly distinguish between Quick Flows (interactive, on-demand workflows) and Quick Automate (scheduled automation with triggers)
– If uncertain about a capability, explicitly state limitations and provide documented alternatives
– Never assume features exist without documentation confirmation
– When correcting previous errors, acknowledge the mistake and provide accurate information based on verified documentation
– Use the documentation knowledgebase available through the attached Space to validate capabilities before making recommendations
C. SOLUTION RECOMMENDATIONS
1. List appropriate Quick Suite capabilities with scalability-matched options:
-For low impact: Start with optimized prompts for default chat agent (My Assistant) or basic Quick Sight BI functionalities as suitable for the use case
-For moderate-high impact: assess and recommend dedicated scalable solutions (aligning with the use case) built as custom chat agent, Flows, Automation projects, required Integrations, Extensions for web browser/Slack/Teams/Outlook/Word specific use cases, relevant Spaces, Research, Quick Sight
-Present multiple options when applicable, prioritizing simplicity when impact doesn’t justify complexity
2. Provide clear reasoning for each suggested capability including:
-Impact-to-complexity analysis
-Scalability considerations (user adoption, maintenance, governance)
-Pros & Cons with emphasis on right-sizing the solution
-Detailed ROI projections including potential time savings multiplied by user count and estimated implementation costs (e.g., “suggested solution would save 7 hours per person across 50 users = 350 hours total weekly savings, equivalent to $X in productivity gains”)
-GenAI adoption benefits and change management considerations
-Prompt engineering best practices for Chat Agents when applicable
3. Ask if they want prescriptive implementation guidance, if they do, then provide detailed solution building pathways including:
-Step-by-step implementation approach starting with minimum viable solution
-Scaling pathway from simple to complex as adoption grows
-Prompt engineering templates and best practices
-GenAI adoption strategies and success metrics
-ROI tracking and measurement recommendations
-Change management recommendations
The three-phase methodology (discovery, analysis, solution recommendations) gives the agent best practices and guidelines on the kind of information it needs to collect to inform its recommendations, so its ability to get data about these features is augmented by user-specified context that is relevant to the recommended solutions.

For Tone, enter a description to calibrate emotional intelligence and approachability:

Professional, consultative, thorough, and evangelistic about GenAI potential while emphasizing practical, right-sized solutions. Ask clarifying questions to ensure accurate recommendations while inspiring confidence in AI adoption without over-engineering.

For Response format, configure the structural patterns (conversational vs. prescriptive, lists vs. paragraphs) that match different interaction phases:

Conversational in DISCOVERY phase with competency and scalability assessment questions. Always ask follow-up questions for clarity before concluding suggestions. Prescriptive in SOLUTION RECOMMENDATIONS phase: Provide structured recommendations with clear reasoning, impact analysis, prompt engineering guidance, and GenAI adoption strategies. Use numbered lists for capabilities and bullet points for implementation details.

For Length, set phase-appropriate boundaries to prevent both overwhelming verbosity and insufficient detail:

Succinct and to-the-point in DISCOVERY phase. For SOLUTION RECOMMENDATIONS phase: Comprehensive enough to cover all relevant Quick Suite capabilities with detailed reasoning, scalability analysis, prompt engineering best practices, and GenAI evangelism insights, but organized for easy scanning.

For Reference documents, you can provide reference documents that give additional guidance to the agent on enterprise considerations and guardrails to keep in mind while recommending solutions, as well as additional nuances about the different features to factor for solution complexity. For this example, we don’t upload additional documents.

For KNOWLEDGE SOURCES:

Choose Link spaces
Choose the Space you created earlier and choose Link.

Linking the Space makes sure the agent can verify capabilities against actual product documentation. The Space architecture maintains enterprise security by honoring underlying data source permissions, allowing AI deployment without compromising existing security permissions. The web crawler option for live documentation makes sure the agent’s knowledge stays current as the platform evolves.

For ACTIONS, set up relevant third-party platform integrations. For example, add one of your enterprise collaboration tools, such as Slack or Teams, for sharing the implementation recommendations from this agent with your team.

Action integrations extend capabilities beyond conversation to actual workflow execution. This dynamic knowledge approach configures an adaptive assistant that validates recommendations against current information, accesses real business data, and executes actions, all while respecting organizational security boundaries.

Update CUSTOMIZATION

For Welcome Message, enter a message such as the following:

Hello! I’m your Quick Suite Product Specialist, GenAI Evangelist, and Pro Prompt Engineer. Let’s DISCOVER your productivity challenge, assess its scalability potential and your GenAI readiness, and I’ll recommend the right-sized SOLUTION that maximizes impact, complete with projected ROI analysis.

For Suggested prompts, enter suggestions that end-users of this chat might use as quick start prompts to talk to the agent:

“What Quick Suite capability can help me with my productivity/automation use case?”
“How can I maximize impact with the simplest possible GenAI solution for my use case?”
“I’m new to GenAI – what’s the best Quick Suite solution to start with for my use case?”

Choose Update preview, test the chat agent, and make adjustments as necessary.
Choose Launch chat agent to publish the agent.
Choose Share to share access to the chat agent as necessary.

Test the chat agent
Let’s demonstrate the capabilities of the Quick Suite Product Specialist that you created:

On the Quick Suite console, choose Chat agents in the navigation pane.
Select the Quick Suite Product Specialist chat agent you created.
On the Actions menu, choose the Chat link.
Send the following request to the agent: “I want to get help in formatting my weekly status emails.”

The agent takes the initial prompts and returns with detailed discovery questionnaire to better understand your use case, without jumping to recommendations. You will notice some differences from run to run, and might not see the same questionnaire, and chat agent responses as shown in the example in this post.

Review and respond to the questionnaire.

The agent returns a comprehensive response including assessment of impact, multiple solution recommendations with reasoning, and high-level implementation pathway options, letting you choose your solution options, and receive prescriptive implementation guidance.

Continue interacting with the agent to get detailed implementation guidance. Try out the chat agent on your own use cases, build out recommended solutions, and learn from your interactions.

Clean up
When you are ready to remove the custom chat agent from your Quick Suite setup, clean up the resources to avoid potential additional indexing costs:

Delete the knowledge base:

On the Quick Suite console, choose Integrations in the navigation pane, then choose Knowledge bases.
Choose the options menu (three dots) next to the knowledge base you created.
Choose Delete knowledge base and follow the prompts to delete the knowledge base.

Delete the Space:

On the Quick Suite console, choose Spaces in the navigation pane.
Choose the options menu (three dots) next to the Space you created.
Choose Delete and follow the prompts to delete the Space.

Delete the chat agent:

On the Quick Suite console, choose Chat agents in the navigation pane.
Choose the options menu (three dots) next to the chat agent you created.
Choose Delete and follow the prompts to delete the chat agent.

Key takeaways
Building effective chat agents requires intentional design across three foundational layers. The Quick Suite Product Specialist demonstrates these principles in action:

Specificity drives consistency – Rather than hoping the LLM will determine the right approach, you can provide explicit identity definitions, behavioral constraints, decision frameworks, and output formats to transform generic AI into reliable expert assistants.
Structure prevents common failures – The three-phase methodology (discovery, analysis, solution recommendations) shows how systematic approaches guide users to right-size solutions, only after understanding the problem.
Dynamic knowledge maintains relevance – Linking live documentation and permission-aware Spaces makes sure agents validate recommendations against current information while respecting organizational security boundaries.

Conclusion
Custom chat agents in Quick Suite can transform how teams access and use enterprise knowledge. By applying the three-layer framework—identity, instructions, and knowledge—you can create AI assistants that deliver instant, accurate answers while maintaining enterprise security and compliance. The Quick Suite Product Specialist example demonstrates how structured methodologies and careful configuration turn generic AI into specialized experts that guide users to the right solutions for their specific needs.
Start with a focused use case that demonstrates clear ROI, then expand as adoption grows. Custom chat agents can deliver measurable productivity gains, helping teams find information faster, automating repetitive workflows, or providing expert guidance at scale. To learn more about creating and deploying Quick Suite chat agents, see Create, customize, and deploy AI-powered chat agents in Amazon Quick Suite.

About the authors
Nitish Chaudhari is a Senior Customer Solutions Manager at AWS, where he partners with customers to architect and implement generative AI solutions. He specializes in building collaborating agents, chat agents, and automation flows with Amazon Quick Suite and Amazon Bedrock that help teams solve real-world productivity challenges at scale. Before joining AWS, Nitish led product teams in the energy sector, and he now works closely with customers and AWS service teams to shape the next generation of generative AI capabilities.
Sindhu Santhanakrishnan is a Senior Product Manager at AWS, where she leads the development of custom agent capabilities in Amazon Quick Suite. She has played a key role in AWS’s automation journey, being part of the Q Apps launch, leading Q Actions in Q Business, and most recently driving the successful launch of chat agents in Quick Suite. She specializes in building business-focused automation solutions, with a background in launching zero-to-one products and customer data platforms. Sindhu holds a Master’s in Product Management from Carnegie Mellon University.
Vinayak Datar is a Senior Solutions Manager based in Bay Area, helping enterprise customers accelerate their AWS Cloud journey. He’s focusing on helping customers to convert ideas from concepts to working prototypes to production using AWS generative AI services.

As AI models grow in complexity and hardware evolves to meet the demand, the software layer connecting the two must also adapt. We recently sat down with Stephen Jones, a Distinguished Engineer at NVIDIA and one of the original architects of CUDA.

Jones, whose background spans from fluid mechanics to aerospace engineering, offered deep insights into NVIDIA’s latest software innovations, including the shift toward tile-based programming, the introduction of “Green Contexts,” and how AI is rewriting the rules of code development.

Here are the key takeaways from our conversation.

The Shift to Tile-Based Abstraction

For years, CUDA programming has revolved around a hierarchy of grids, blocks, and threads. With the latest updates, NVIDIA is introducing a higher level of abstraction: CUDA Tile.

According to Jones, this new approach allows developers to program directly to arrays and tensors rather than managing individual threads. “It extends the existing CUDA,” Jones explained. “What we’ve done is we’ve added a way to talk about and program directly to arrays, tensors, vectors of data… allowing the language and the compiler to see what the high-level data was that you’re operating on opened up a whole realm of new optimizations”.

This shift is partly a response to the rapid evolution of hardware. As Tensor Cores become larger and denser to combat the slowing of Moore’s Law, the mapping of code to silicon becomes increasingly complex.

Future-Proofing: Jones noted that by expressing programs as vector operations (e.g., Tensor A times Tensor B), the compiler takes on the heavy lifting of mapping data to the specific hardware generation.

Stability: This ensures that program structure remains stable even as the underlying GPU architecture changes from Ampere to Hopper to Blackwell.

Python First, But Not Python Only

Recognizing that Python has become the lingua franca of Artificial Intelligence, NVIDIA launched CUDA Tile support with Python first. “Python’s the language of AI,” Jones stated, adding that an array-based representation is “much more natural to Python programmers” who are accustomed to NumPy.

However, performance purists need not worry. C++ support is arriving next year, maintaining NVIDIA’s philosophy that developers should be able to accelerate their code regardless of the language they choose.

“Green Contexts” and Reducing Latency

For engineers deploying Large Language Models (LLMs) in production, latency and jitter are critical concerns. Jones highlighted a new feature called Green Contexts, which allows for precise partitioning of the GPU.

“Green contexts lets you partition the GPU… into different sections,” Jones said. This allows developers to dedicate specific fractions of the GPU to different tasks, such as running pre-fill and decode operations simultaneously without them competing for resources. This micro-level specialization within a single GPU mirrors the disaggregation seen at the data center scale.

No Black Boxes: The Importance of Tooling

One of the pervasive fears regarding high-level abstractions is the loss of control. Jones, drawing on his experience as a CUDA user in the aerospace industry, emphasized that NVIDIA tools will never be black boxes.

“I really believe that the most important part of CUDA is the developer tools,” Jones affirmed. He assured developers that even when using tile-based abstractions, tools like Nsight Compute will allow inspection down to the individual machine language instructions and registers. “You’ve got to be able to tune and debug and optimize… it cannot be a black box,” he added.

Accelerating Time-to-Result

Ultimately, the goal of these updates is productivity. Jones described the objective as “left shifting” the performance curve, enabling developers to reach 80% of potential performance in a fraction of the time.

“If you can come to market [with] 80% of performance in a week instead of a month… then you’re spending the rest of your time just optimizing,” Jones explained. Crucially, this ease of use does not come at the cost of power; the new model still provides a path to 100% of the peak performance the silicon can offer.

Conclusion

As AI algorithms and scientific computing converge, NVIDIA is positioning CUDA not just as a low-level tool for hardware experts, but as a flexible platform that adapts to the needs of Python developers and HPC researchers alike. With support extending from Ampere to the upcoming Blackwell and Rubin architectures, these updates promise to streamline development across the entire GPU ecosystem.

For the full technical details on CUDA Tile and Green Contexts, visit the NVIDIA developer portal.
The post Interview: From CUDA to Tile-Based Programming: NVIDIA’s Stephen Jones on Building the Future of AI appeared first on MarkTechPost.