Category: Uncategorized

In this tutorial, we explore LitServe, a lightweight and powerful serving framework that allows us to deploy machine learning models as APIs with minimal effort. We build and test multiple endpoints that demonstrate real-world functionalities such as text generation, batching, streaming, multi-task processing, and caching, all running locally without relying on external APIs. By the end, we clearly understand how to design scalable and flexible ML serving pipelines that are both efficient and easy to extend for production-level applications. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browser!pip install litserve torch transformers -q

import litserve as ls
import torch
from transformers import pipeline
import time
from typing import List

We begin by setting up our environment on Google Colab and installing all required dependencies, including LitServe, PyTorch, and Transformers. We then import the essential libraries and modules that will allow us to define, serve, and test our APIs efficiently. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserclass TextGeneratorAPI(ls.LitAPI):
def setup(self, device):
self.model = pipeline(“text-generation”, model=”distilgpt2″, device=0 if device == “cuda” and torch.cuda.is_available() else -1)
self.device = device
def decode_request(self, request):
return request[“prompt”]
def predict(self, prompt):
result = self.model(prompt, max_length=100, num_return_sequences=1, temperature=0.8, do_sample=True)
return result[0][‘generated_text’]
def encode_response(self, output):
return {“generated_text”: output, “model”: “distilgpt2”}

class BatchedSentimentAPI(ls.LitAPI):
def setup(self, device):
self.model = pipeline(“sentiment-analysis”, model=”distilbert-base-uncased-finetuned-sst-2-english”, device=0 if device == “cuda” and torch.cuda.is_available() else -1)
def decode_request(self, request):
return request[“text”]
def batch(self, inputs: List[str]) -> List[str]:
return inputs
def predict(self, batch: List[str]):
results = self.model(batch)
return results
def unbatch(self, output):
return output
def encode_response(self, output):
return {“label”: output[“label”], “score”: float(output[“score”]), “batched”: True}

Here, we create two LitServe APIs, one for text generation using a local DistilGPT2 model and another for batched sentiment analysis. We define how each API decodes incoming requests, performs inference, and returns structured responses, demonstrating how easy it is to build scalable, reusable model-serving endpoints. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserclass StreamingTextAPI(ls.LitAPI):
def setup(self, device):
self.model = pipeline(“text-generation”, model=”distilgpt2″, device=0 if device == “cuda” and torch.cuda.is_available() else -1)
def decode_request(self, request):
return request[“prompt”]
def predict(self, prompt):
words = [“Once”, “upon”, “a”, “time”, “in”, “a”, “digital”, “world”]
for word in words:
time.sleep(0.1)
yield word + ” ”
def encode_response(self, output):
for token in output:
yield {“token”: token}

In this section, we design a streaming text-generation API that emits tokens as they are generated. We simulate real-time streaming by yielding words one at a time, demonstrating how LitServe can handle continuous token generation efficiently. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserclass MultiTaskAPI(ls.LitAPI):
def setup(self, device):
self.sentiment = pipeline(“sentiment-analysis”, device=-1)
self.summarizer = pipeline(“summarization”, model=”sshleifer/distilbart-cnn-6-6″, device=-1)
self.device = device
def decode_request(self, request):
return {“task”: request.get(“task”, “sentiment”), “text”: request[“text”]}
def predict(self, inputs):
task = inputs[“task”]
text = inputs[“text”]
if task == “sentiment”:
result = self.sentiment(text)[0]
return {“task”: “sentiment”, “result”: result}
elif task == “summarize”:
if len(text.split()) < 30:
return {“task”: “summarize”, “result”: {“summary_text”: text}}
result = self.summarizer(text, max_length=50, min_length=10)[0]
return {“task”: “summarize”, “result”: result}
else:
return {“task”: “unknown”, “error”: “Unsupported task”}
def encode_response(self, output):
return output

We now develop a multi-task API that handles both sentiment analysis and summarization via a single endpoint. This snippet demonstrates how we can manage multiple model pipelines through a unified interface, dynamically routing each request to the appropriate pipeline based on the specified task. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserclass CachedAPI(ls.LitAPI):
def setup(self, device):
self.model = pipeline(“sentiment-analysis”, device=-1)
self.cache = {}
self.hits = 0
self.misses = 0
def decode_request(self, request):
return request[“text”]
def predict(self, text):
if text in self.cache:
self.hits += 1
return self.cache[text], True
self.misses += 1
result = self.model(text)[0]
self.cache[text] = result
return result, False
def encode_response(self, output):
result, from_cache = output
return {“label”: result[“label”], “score”: float(result[“score”]), “from_cache”: from_cache, “cache_stats”: {“hits”: self.hits, “misses”: self.misses}}

We implement an API that uses caching to store previous inference results, reducing redundant computation for repeated requests. We track cache hits and misses in real time, illustrating how simple caching mechanisms can drastically improve performance in repeated inference scenarios. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserdef test_apis_locally():
print(“=” * 70)
print(“Testing APIs Locally (No Server)”)
print(“=” * 70)

api1 = TextGeneratorAPI(); api1.setup(“cpu”)
decoded = api1.decode_request({“prompt”: “Artificial intelligence will”})
result = api1.predict(decoded)
encoded = api1.encode_response(result)
print(f”✓ Result: {encoded[‘generated_text’][:100]}…”)

api2 = BatchedSentimentAPI(); api2.setup(“cpu”)
texts = [“I love Python!”, “This is terrible.”, “Neutral statement.”]
decoded_batch = [api2.decode_request({“text”: t}) for t in texts]
batched = api2.batch(decoded_batch)
results = api2.predict(batched)
unbatched = api2.unbatch(results)
for i, r in enumerate(unbatched):
encoded = api2.encode_response(r)
print(f”✓ ‘{texts[i]}’ -> {encoded[‘label’]} ({encoded[‘score’]:.2f})”)

api3 = MultiTaskAPI(); api3.setup(“cpu”)
decoded = api3.decode_request({“task”: “sentiment”, “text”: “Amazing tutorial!”})
result = api3.predict(decoded)
print(f”✓ Sentiment: {result[‘result’]}”)

api4 = CachedAPI(); api4.setup(“cpu”)
test_text = “LitServe is awesome!”
for i in range(3):
decoded = api4.decode_request({“text”: test_text})
result = api4.predict(decoded)
encoded = api4.encode_response(result)
print(f”✓ Request {i+1}: {encoded[‘label’]} (cached: {encoded[‘from_cache’]})”)

print(“=” * 70)
print(” All tests completed successfully!”)
print(“=” * 70)

test_apis_locally()

We test all our APIs locally to verify their correctness and performance without starting an external server. We sequentially evaluate text generation, batched sentiment analysis, multi-tasking, and caching, ensuring each component of our LitServe setup runs smoothly and efficiently.

In conclusion, we create and run diverse APIs that showcase the framework’s versatility. We experiment with text generation, sentiment analysis, multi-tasking, and caching to experience LitServe’s seaMLess integration with Hugging Face pipelines. As we complete the tutorial, we realize how LitServe simplifies model deployment workflows, enabling us to serve intelligent ML systems in just a few lines of Python code while maintaining flexibility, performance, and simplicity.

Check out the FULL CODES here. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post An Implementation on Building Advanced Multi-Endpoint Machine Learning APIs with LitServe: Batching, Streaming, Caching, and Local Inference appeared first on MarkTechPost.

A team of Salesforce AI researchers introduced WALT (Web Agents that Learn Tools), a framework that reverse-engineers latent website functionality into reusable invocable tools. It reframes browser automation around callable tools rather than long chains of clicks. Agents then call operations such as search, filter, sort, post_comment, and create_listing. This reduces dependence on large language model step by step reasoning and increases determinism during execution.

https://arxiv.org/pdf/2510.01524

What WALT builds?

Web agents often fail when layouts shift or when tasks require long sequences. WALT targets this failure mode by mining site functionality offline, then exposing it as tools that encapsulate navigation, selection, extraction, and optional agentic steps. Tools carry contracts in the form of schemas and examples. At runtime, an agent composes a short program with a few tool calls to complete a task. The design goal is higher success with fewer steps and less reliance on free form reasoning.

Pipeline in two phases

The pipeline has discovery and construction with validation. In discovery, WALT explores a website and proposes tool candidates that map to common goals such as discovery, content management, and communication. In construction and validation, WALT converts traces to deterministic scripts, stabilizes selectors, attempts URL promotion when possible, induces an input schema, and registers a tool only after end to end checks pass. This shifts as much work as possible into stable URL and form operations and leaves agentic grounding for the cases that truly require it.

https://arxiv.org/pdf/2510.01524

Results on VisualWebArena and WebArena

On VisualWebArena, WALT reports an average success rate of 52.9 percent with per split results of 64.1 percent on Classifieds, 53.4 percent on Shopping, and 39.0 percent on Reddit. The table lists baselines such as SGV at 50.2 percent and ExaCT at 33.7 percent. Human performance is 88.7 percent on average.

On WebArena, WALT reaches 50.1 percent average across GitLab, Map, Shopping, CMS, Reddit, and Multi. The table shows WALT ahead of prior methods with a nine point margin over the best skill induction baseline. Human performance is 78.2 percent.

https://arxiv.org/pdf/2510.01524

Efficiency and ablations

Tools reduce action count by a factor near 1.4 on average relative to a matched agent without tools. On the Classifieds split, ablations show consistent gains when tools are used across different agent backbones. WALT with GPT 5 mini records 7 percent higher success and 27 percent fewer steps, while a human demonstration strategy yields 66.0 percent success. The fully autonomous WALT reaches 64.1 percent with 5 percent fewer steps than the human demonstration case. Multimodal DOM parsing adds 2.6 percent absolute improvement. External verification adds 3.3 percent while increasing checks. Across components, WALT records 21.3 percent fewer steps than baseline policies.

https://arxiv.org/pdf/2510.01524

Design choices that enforce determinism

WALT prefers URL level operations when the site exposes query parameters or routes for search and filtering. When pages require dynamic grounding, the tool script inserts bounded agentic steps such as content extraction or wait for page load. Selector stabilization and schema validation reduce drift when sites change. The method keeps the fraction of agentic operations low in discovered tool sets and biases toward deterministic actions like navigation, input, and click.

Key Takeaways

Approach: WALT discovers and validates website-native functions, then exposes them as callable tools with input schemas, selector stabilization, and URL promotion, reducing brittle step sequences to deterministic operations.

Results — VisualWebArena: Average success rate 52.9%, with 64.1% on Classifieds, 53.4% on Shopping, and 39.0% on Reddit, outperforming several baselines reported in the paper.

Results — WebArena: Average success rate 50.1% across GitLab, Map, Shopping, CMS, Reddit, and Multi, showing consistent gains over skill-induction and search-based baselines.

Efficiency and Ablations: Toolization cuts steps by about 1.4x, with 21.3% fewer actions on average. Multimodal DOM parsing adds +2.6% absolute success, and external verification adds +3.3%.

Editorial Comments

WALT is a useful pivot from step sequence agents to functionality grounded tools. The framework reverse engineers latent website functionality into reusable invocable tools across discovery, content management, and communication. By promoting UI traces to deterministic tools with schema validation and URL operations, WALT lifts web agent success to 52.9 percent on VisualWebArena and 50.1 percent on WebArena, while cutting actions by about 21.3 percent. The release ships a CLI, walt discover, walt agent, and MCP serving for integration.

Check out the Paper and GitHub Page. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post Salesforce AI Research Introduces WALT (Web Agents that Learn Tools): Enabling LLM agents to Automatically Discover Reusable Tools from Any Website appeared first on MarkTechPost.

Generative AI has emerged as a transformative technology in healthcare, driving digital transformation in essential areas such as patient engagement and care management. It has shown potential to revolutionize how clinicians provide improved care through automated systems with diagnostic support tools that provide timely, personalized suggestions, ultimately leading to better health outcomes. For example, a study reported in BMC Medical Education that medical students who received large language model (LLM)-generated feedback during simulated patient interactions significantly improved their clinical decision-making compared to those who did not.
At the center of most generative AI systems are LLMs capable of generating remarkably natural conversations, enabling healthcare customers to build products across billing, diagnosis, treatment, and research that can perform tasks and operate independently with human oversight. However, the utility of generative AI requires an understanding of the potential risks and impacts on healthcare service delivery, which necessitates the need for careful planning, definition, and execution of a system-level approach to building safe and responsible generative AI-infused applications.
In this post, we focus on the design phase of building healthcare generative AI applications, including defining system-level policies that determine the inputs and outputs. These policies can be thought of as guidelines that, when followed, help build a responsible AI system.
Designing responsibly
LLMs can transform healthcare by reducing the cost and time required for considerations such as quality and reliability. As shown in the following diagram, responsible AI considerations can be successfully integrated into an LLM-powered healthcare application by considering quality, reliability, trust, and fairness for everyone. The goal is to promote and encourage certain responsible AI functionalities of AI systems. Examples include the following:

Each component’s input and output is aligned with clinical priorities to maintain alignment and promote controllability
Safeguards, such as guardrails, are implemented to enhance the safety and reliability of your AI system
Comprehensive AI red-teaming and evaluations are applied to the entire end-to-end system to assess safety and privacy-impacting inputs and outputs

Conceptual architecture
The following diagram shows a conceptual architecture of a generative AI application with an LLM. The inputs (directly from an end-user) are mediated through input guardrails. After the input has been accepted, the LLM can process the user’s request using internal data sources. The output of the LLM is again mediated through guardrails and can be shared with end-users.

Establish governance mechanisms
When building generative AI applications in healthcare, it’s essential to consider the various risks at the individual model or system level, as well as at the application or implementation level. The risks associated with generative AI can differ from or even amplify existing AI risks. Two of the most important risks are confabulation and bias:

Confabulation — The model generates confident but erroneous outputs, sometimes referred to as hallucinations. This could mislead patients or clinicians.
Bias — This refers to the risk of exacerbating historical societal biases among different subgroups, which can result from non-representative training data.

To mitigate these risks, consider establishing content policies that clearly define the types of content your applications should avoid generating. These policies should also guide how to fine-tune models and which appropriate guardrails to implement. It is crucial that the policies and guidelines are tailored and specific to the intended use case. For instance, a generative AI application designed for clinical documentation should have a policy that prohibits it from diagnosing diseases or offering personalized treatment plans.
Additionally, defining clear and detailed policies that are specific to your use case is fundamental to building responsibly. This approach fosters trust and helps developers and healthcare organizations carefully consider the risks, benefits, limitations, and societal implications associated with each LLM in a particular application.
The following are some example policies you might consider using for your healthcare-specific applications. The first table summarizes the roles and responsibilities for human-AI configurations.

Action ID
Suggested Action
Generative AI Risks

GV-3.2-001
Policies are in place to bolster oversight of generative AI systems with independent evaluations or assessments of generative AI models or systems where the type and robustness of evaluations are proportional to the identified risks.
CBRN Information or Capabilities; Harmful Bias and Homogenization

GV-3.2-002
Consider adjustment of organizational roles and components across lifecycle stages of large or complex generative AI systems, including: test and evaluation, validation, and red-teaming of generative AI systems; generative AI content moderation; generative AI system development and engineering; increased accessibility of generative AI tools, interfaces, and systems; and incident response and containment.
Human-AI Configuration; Information Security; Harmful Bias and Homogenization

GV-3.2-003
Define acceptable use policies for generative AI interfaces, modalities, and human-AI configurations (for example, for AI assistants and decision-making tasks), including criteria for the kinds of queries generative AI applications should refuse to respond to.
Human-AI Configuration

GV-3.2-004
Establish policies for user feedback mechanisms for generative AI systems that include thorough instructions and any mechanisms for recourse.
Human-AI Configuration

GV-3.2-005
Engage in threat modeling to anticipate potential risks from generative AI systems.
CBRN Information or Capabilities; Information Security

The following table summarizes policies for risk management in AI system design.

Action ID
Suggested Action
Generative AI Risks

GV-4.1-001
Establish policies and procedures that address continual improvement processes for generative AI risk measurement. Address general risks associated with a lack of explainability and transparency in generative AI systems by using ample documentation and techniques such as application of gradient-based attributions, occlusion or term reduction, counterfactual prompts and prompt engineering, and analysis of embeddings. Assess and update risk measurement approaches at regular cadences.
Confabulation

GV-4.1-002
Establish policies, procedures, and processes detailing risk measurement in context of use with standardized measurement protocols and structured public feedback exercises such as AI red-teaming or independent external evaluations.
CBRN Information and Capability; Value Chain and Component Integration

Transparency artifacts
Promoting transparency and accountability throughout the AI lifecycle can foster trust, facilitate debugging and monitoring, and enable audits. This involves documenting data sources, design decisions, and limitations through tools like model cards and offering clear communication about experimental features. Incorporating user feedback mechanisms further supports continuous improvement and fosters greater confidence in AI-driven healthcare solutions.
AI developers and DevOps engineers should be transparent about the evidence and reasons behind all outputs by providing clear documentation of the underlying data sources and design decisions so that end-users can make informed decisions about the use of the system. Transparency enables the tracking of potential problems and facilitates the evaluation of AI systems by both internal and external teams. Transparency artifacts guide AI researchers and developers on the responsible use of the model, promote trust, and help end-users make informed decisions about the use of the system.
The following are some implementation suggestions:

When building AI features with experimental models or services, it’s essential to highlight the possibility of unexpected model behavior so healthcare professionals can accurately assess whether to use the AI system.
Consider publishing artifacts such as Amazon SageMaker model cards or AWS system cards. Also, at AWS we provide detailed information about our AI systems through AWS AI Service Cards, which list intended use cases and limitations, responsible AI design choices, and deployment and performance optimization best practices for some of our AI services. AWS also recommends establishing transparency policies and processes for documenting the origin and history of training data while balancing the proprietary nature of training approaches. Consider creating a hybrid document that combines elements of both model cards and service cards, because your application likely uses foundation models (FMs) but provides a specific service.
Offer a feedback user mechanism. Gathering regular and scheduled feedback from healthcare professionals can help developers make necessary refinements to improve system performance. Also consider establishing policies to help developers allow for user feedback mechanisms for AI systems. These should include thorough instructions and consider establishing policies for any mechanisms for recourse.

Security by design
When developing AI systems, consider security best practices at each layer of the application. Generative AI systems might be vulnerable to adversarial attacks suck as prompt injection, which exploits the vulnerability of LLMs by manipulating their inputs or prompt. These types of attacks can result in data leakage, unauthorized access, or other security breaches. To address these concerns, it can be helpful to perform a risk assessment and implement guardrails for both the input and output layers of the application. As a general rule, your operating model should be designed to perform the following actions:

Safeguard patient privacy and data security by implementing personally identifiable information (PII) detection, configuring guardrails that check for prompt attacks
Continually assess the benefits and risks of all generative AI features and tools and regularly monitor their performance through Amazon CloudWatch or other alerts
Thoroughly evaluate all AI-based tools for quality, safety, and equity before deploying

Developer resources
The following resources are useful when architecting and building generative AI applications:

Amazon Bedrock Guardrails helps you implement safeguards for your generative AI applications based on your use cases and responsible AI policies. You can create multiple guardrails tailored to different use cases and apply them across multiple FMs, providing a consistent user experience and standardizing safety and privacy controls across your generative AI applications.
The AWS responsible AI whitepaper serves as an invaluable resource for healthcare professionals and other developers that are developing AI applications in critical care environments where errors could have life-threatening consequences.
AWS AI Service Cards explains the use cases for which the service is intended, how machine learning (ML) is used by the service, and key considerations in the responsible design and use of the service.

Conclusion
Generative AI has the potential to improve nearly every aspect of healthcare by enhancing care quality, patient experience, clinical safety, and administrative safety through responsible implementation. When designing, developing, or operating an AI application, try to systematically consider potential limitations by establishing a governance and evaluation framework grounded by the need to maintain the safety, privacy, and trust that your users expect.
For more information about responsible AI, refer to the following resources:

NIST Trustworthy and Responsible AI
OWASP Top 10 for Large Language Model applications

About the authors
Tonny Ouma is an Applied AI Specialist at AWS, specializing in generative AI and machine learning. As part of the Applied AI team, Tonny helps internal teams and AWS customers incorporate leading-edge AI systems into their products. In his spare time, Tonny enjoys riding sports bikes, golfing, and entertaining family and friends with his mixology skills.
Simon Handley, PhD, is a Senior AI/ML Solutions Architect in the Global Healthcare and Life Sciences team at Amazon Web Services. He has more than 25 years’ experience in biotechnology and machine learning and is passionate about helping customers solve their machine learning and life sciences challenges. In his spare time, he enjoys horseback riding and playing ice hockey.

Open vocabulary object detectors answer text queries with boxes. In remote sensing, zero shot performance drops because classes are fine grained and visual context is unusual. Google Research team proposess FLAME, a one step active learning strategy that rides on a strong open vocabulary detector and adds a tiny refiner that you can train in near real time on a CPU. The base model generates high recall proposals, the refiner filters false positives with a few targeted labels, and you avoid full model fine tuning. It reports state of the art accuracy on DOTA and DIOR with 30 shots, and minute scale adaptation per label on a CPU.

https://arxiv.org/pdf/2510.17670v1

Problem framing

Open vocabulary detectors such as OWL ViT v2 are trained on web scale image text pairs. They generalize well on natural images, yet they struggle when categories are subtle, for example chimney versus storage tank, or when the imaging geometry is different, for example nadir aerial tiles with rotated objects and small scales. Precision falls because the text embedding and the visual embedding overlap for look alike categories. A practical system needs the breadth of open vocabulary models, and the precision of a local specialist, without hours of GPU fine tuning or thousands of new labels.

Method and design in concise

FLAME is a cascaded pipeline. Step one, run a zero shot open vocabulary detector to produce many candidate boxes for a text query, for example “chimney.” Step two, represent each candidate with visual features and its similarity to the text. Step three, retrieve marginal samples that sit near the decision boundary by doing a low dimensional projection with PCA, then a density estimate, then select the uncertain band. Step four, cluster this band and pick one item per cluster for diversity. Step five, have a user label about 30 crops as positive or negative. Step six, optionally rebalance with SMOTE or SVM SMOTE if the labels are skewed. Step seven, train a small classifier, for example an RBF SVM or a two layer MLP, to accept or reject the original proposals. The base detector stays frozen, so you keep recall and generalization, and the refiner learns the exact semantics the user meant.

https://arxiv.org/pdf/2510.17670v1

Datasets, base models, and setup

Evaluation uses two standard remote sensing detection benchmarks. DOTA has oriented boxes over 15 categories in high resolution aerial images. DIOR has 23,463 images and 192,472 instances over 20 categories. The comparison includes a zero shot OWL ViT v2 baseline, a zero shot RS OWL ViT v2 that is fine tuned on RS WebLI, and several few shot baselines. RS OWL ViT v2 improves zero shot mean AP to 31.827 percent on DOTA and 29.387 percent on DIOR, which becomes the starting point for FLAME.

https://arxiv.org/pdf/2510.17670v1

Understanding the Results

On 30 shot adaptation, FLAME cascaded on RS OWL ViT v2 reaches 53.96 percent AP on DOTA and 53.21 percent AP on DIOR, which is the top accuracy among the listed methods. The comparison includes SIoU, a prototype based method with DINOv2, and a few shot method proposed by the research team. These numbers appear in Table 1. The research team also reports the per class breakdown in Table 2. On DIOR, the chimney class improves from 0.11 in zero shot to 0.94 after FLAME, which illustrates how the refiner removes look alike false positives from the open vocabulary proposals.

https://arxiv.org/pdf/2510.17670v1

Key Takeaways

FLAME is a one step active learning cascade over OWL ViT v2, it retrieves marginal samples using density estimation, enforces diversity with clustering, collects about 30 labels, and trains a lightweight refiner such as an RBF SVM or a small MLP, with no base model fine tuning.

With 30 shots, FLAME on RS OWL ViT v2 reaches 53.96% AP on DOTA and 53.21% AP on DIOR, exceeding prior few shot baselines including SIoU and a prototype method with DINOv2.

On DIOR, the chimney class improves from 0.11 in zero shot to 0.94 after FLAME, which shows strong filtering of look alike false positives.

Adaptation runs in about 1 minute for each label on a standard CPU, which supports near real time, user in the loop specialization.

Zero shot OWL ViT v2 starts at 13.774% AP on DOTA and 14.982% on DIOR, RS OWL ViT v2 raises zero shot AP to 31.827% and 29.387% respectively, and FLAME then delivers the large precision gains on top.

Editorial Comments

FLAME is a one step active learning cascade that layers a tiny refiner on top of OWL ViT v2, selecting marginal detections, collecting about 30 labels, and training a small classifier without touching the base model. On DOTA and DIOR, FLAME with RS OWL ViT v2 reports 53.96 percent AP and 53.21 percent AP, establishing a strong few shot baseline. On DIOR chimney, average precision rises from 0.11 to 0.94 after refinement, illustrating false positive suppression. Adaptation runs in about 1 minute per label on a CPU, enabling interactive specialization. OWLv2 and RS WebLI provide the foundation for zero shot proposals. Overall, FLAME demonstrates a practical path to open vocabulary detection specialization in remote sensing by pairing RS OWL ViT v2 proposals with a minute scale CPU refiner that lifts DOTA to 53.96 percent AP and DIOR to 53.21 percent AP.

Check out the Paper here. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post Google AI Introduces FLAME Approach: A One-Step Active Learning that Selects the Most Informative Samples for Training and Makes a Model Specialization Super Fast appeared first on MarkTechPost.

Computer-use agents have been limited to primitives. They click, they type, they scroll. Long action chains amplify grounding errors and waste steps. Apple Researchers introduce UltraCUA, a foundation model that builds an hybrid action space that lets an agent interleave low level GUI actions with high level programmatic tool calls. The model chooses the cheaper and more reliable move at each step. The approach improves success and reduces steps on OSWorld, and transfers to WindowsAgentArena without Windows specific training.

https://arxiv.org/pdf/2510.17790

What hybrid action changes?

Hybrid action treats tools as first class actions. A tool call encapsulates a multi step operation as a single function with a clear signature and a docstring. A click or a key press still exists when no programmatic path is available. The agent learns to alternate between both modes. The goal is to reduce cascade errors and to cut step counts. The research team positions this as a bridge between GUI only CUAs and tool centric agent frameworks.

https://arxiv.org/pdf/2510.17790

Scaled tool acquisition

UltraCUA builds its tool library with an automated pipeline. The system extracts keyboard shortcuts and commands from software documentation. The system integrates open source implementations from agent toolkits. The system also uses coding agents to synthesize new tools. Each tool is a callable interface that hides a long GUI sequence. The research team reports coverage across 10 desktop domains with 881 tools. The largest buckets include VS Code with 135 tools and LibreOffice Writer with 123 tools. Thunderbird and GIMP also have deep coverage.

https://arxiv.org/pdf/2510.17790

Verifiable synthetic tasks and trajectories

Training requires grounded supervision and stable rewards. UltraCUA uses a dual synthetic engine. An evaluator first pipeline composes atomic verifiers for browsers, files, images, and system state, then generates tasks that satisfy those checks. An instruction first pipeline explores the OS and proposes context aligned tasks which are then verified. The result is 17,864 verifiable tasks across 10 domains such as Chrome, LibreOffice, GIMP, VS Code, system, Thunderbird, VLC, and multi app workflows. Chrome has 2,826 tasks. The LibreOffice suite sums to 5,885 tasks. Multi app tasks reach 2,113.

https://arxiv.org/pdf/2510.17790

A multi agent rollout produces successful hybrid trajectories. The planner uses OpenAI o3 for decision making. The grounder uses GTA1-7B for accurate visual localization. The rollout yields about 26.8K successful trajectories that show when to use a tool and when to act in the GUI. These trajectories are the core of the supervised phase.

Training Approach

Training has two stages. Stage 1 is supervised fine tuning. The models train for 3 epochs at a learning rate of 2e-5 on the successful trajectories. Loss is applied turn wise to avoid over weighting early steps. Stage 2 is online reinforcement learning. The models train for 150 steps at a learning rate of 1e-6 on verified tasks that are sampled by difficulty. The policy optimization follows a GRPO variant with clip higher, and removes KL regularization and format rewards. The reward combines sparse task outcome with a tool use term. Experiments use NVIDIA H100 GPUs. The context is kept near 32K by controlling the number of exposed tools.

Results on OSWorld

UltraCUA improves success at both 7B and 32B scales. Under 15 step budgets, UltraCUA-32B reaches 41.0 percent success. OpenCUA-32B reaches 29.7 percent. The absolute gain is 11.3 points. UltraCUA-7B reaches 28.9 percent. UI-TARS-1.5-7B reaches 23.4 percent. Gains persist under 50 step budgets. A per domain breakdown shows consistent lifts across Chrome, Writer, VS Code, and cross application tasks. Average steps decrease against baselines. These shifts indicate better action selection rather than only more attempts.

https://arxiv.org/pdf/2510.17790

https://arxiv.org/pdf/2510.17790

Cross platform transfer on WindowsAgentArena

UltraCUA trains only on Ubuntu based OSWorld data. The model is then evaluated on WindowsAgentArena. UltraCUA-7B reaches 21.7 percent success. This exceeds UI-TARS-1.5-7B at 18.1 percent and a Qwen2 baseline trained with Windows data at 13.5 percent. The result suggests that hybrid action strategies learned on one platform transfer to other platforms. The paper highlights this as zero shot platform generalization.

https://arxiv.org/pdf/2510.17790

Key Takeaways

UltraCUA formalizes a hybrid action space that lets a single agent alternate between GUI primitives and programmatic tool calls, which reduces long error prone action chains.

The research team scales a reusable tool library through an automated pipeline and pairs it with a synthetic data engine, yielding 17,000 plus verifiable computer use tasks for training and evaluation.

Training follows a two stage recipe, supervised fine tuning on successful hybrid trajectories then online reinforcement learning on verifiable tasks, which optimizes when to call tools versus act in the GUI.

On OSWorld, UltraCUA reports an average 22 percent relative improvement over base models and 11 percent fewer steps, which indicates gains in reliability and efficiency.

The 7B model reaches 21.7 percent success on WindowsAgentArena without Windows specific training, which shows cross platform generalization of the hybrid action policy.

Editorial Comments

UltraCUA moves computer use agents from brittle primitive action chains to a hybrid action policy, integrating GUI primitives with programmatic tool calls, which reduces error propagation and step counts. It scales tools via an automated pipeline and pairs them with a synthetic data engine that yields 17,000 plus verifiable tasks, enabling supervised fine tuning and online reinforcement learning on grounded signals. Reported results include 22 percent relative improvement on OSWorld with 11 percent fewer steps, and 21.7 percent success on WindowsAgentArena without Windows specific training, which indicates cross platform transfer of the policy.

Check out the Paper here. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post UltraCUA: A Foundation Computer-Use Agents Model that Bridges the Gap between General-Purpose GUI Agents and Specialized API-based Agents appeared first on MarkTechPost.

In this tutorial, we explore how to build a small yet functional multi-agent system using the uAgents framework. We set up three agents — Directory, Seller, and Buyer — that communicate via well-defined message protocols to simulate a real-world marketplace interaction. We design message schemas, define agent behaviors, and implement request-response cycles to demonstrate discovery, negotiation, and transaction among agents, all running asynchronously in a shared event loop. Through this, we understand how autonomous agents collaborate, trade, and efficiently maintain decentralized workflows. Check out the Full Codes here.

Copy CodeCopiedUse a different Browser!pip -q install “uagents>=0.11.2″

import asyncio, random
from typing import List, Dict, Optional
from uagents import Agent, Context, Bureau, Model, Protocol

class ServiceAnnounce(Model):
category: str
endpoint: str

class ServiceQuery(Model):
category: str

class ServiceList(Model):
addresses: List[str]

class OfferRequest(Model):
item: str
max_price: int

class Offer(Model):
item: str
price: int
qty: int

class Order(Model):
item: str
qty: int

class Receipt(Model):
item: str
qty: int
total: int
ok: bool
note: Optional[str] = None

We begin by installing the uAgents library and defining all the message models that underpin our communication system. We create structured data types for announcements, queries, offers, and orders, enabling agents to exchange information seamlessly. Check out the Full Codes here.

Copy CodeCopiedUse a different Browserregistry_proto = Protocol(name=”registry”, version=”1.0″)
trade_proto = Protocol(name=”trade”, version=”1.0″)

directory = Agent(name=”directory”, seed=”dir-seed-001″)
seller = Agent(name=”seller”, seed=”seller-seed-001″)
buyer = Agent(name=”buyer”, seed=”buyer-seed-001″)

directory.include(registry_proto)
seller.include(trade_proto)
buyer.include(registry_proto)
buyer.include(trade_proto)

@registry_proto.on_message(model=ServiceAnnounce)
async def on_announce(ctx: Context, sender: str, msg: ServiceAnnounce):
reg = await ctx.storage.get(“reg”) or {}
reg.setdefault(msg.category, set()).add(sender)
await ctx.storage.set(“reg”, reg)
ctx.logger.info(f”Registered {sender} under ‘{msg.category}'”)

@registry_proto.on_message(model=ServiceQuery)
async def on_query(ctx: Context, sender: str, msg: ServiceQuery):
reg = await ctx.storage.get(“reg”) or {}
addrs = sorted(list(reg.get(msg.category, set())))
await ctx.send(sender, ServiceList(addresses=addrs))
ctx.logger.info(f”Returned {len(addrs)} providers for ‘{msg.category}'”)

We set up the Directory, Seller, and Buyer agents and define the registry protocol that manages service discovery. We make the directory respond to announcements and queries, allowing agents to register and locate each other dynamically. Check out the Full Codes here.

Copy CodeCopiedUse a different BrowserCATALOG: Dict[str, Dict[str, int]] = {
“camera”: {“price”: 120, “qty”: 3},
“laptop”: {“price”: 650, “qty”: 2},
“headphones”: {“price”: 60, “qty”: 5},
}

@seller.on_event(“startup”)
async def seller_start(ctx: Context):
await ctx.send(directory.address, ServiceAnnounce(category=”electronics”, endpoint=seller.address))
ctx.logger.info(“Seller announced to directory”)

@trade_proto.on_message(model=OfferRequest)
async def on_offer_request(ctx: Context, sender: str, req: OfferRequest):
item = CATALOG.get(req.item)
if not item:
await ctx.send(sender, Offer(item=req.item, price=0, qty=0))
return
price = max(1, int(item[“price”] * (0.9 + 0.2 * random.random())))
if price > req.max_price or item[“qty”] <= 0:
await ctx.send(sender, Offer(item=req.item, price=0, qty=0))
return
await ctx.send(sender, Offer(item=req.item, price=price, qty=item[“qty”]))
ctx.logger.info(f”Offered {req.item} at {price} with qty {item[‘qty’]}”)

@trade_proto.on_message(model=Order)
async def on_order(ctx: Context, sender: str, order: Order):
item = CATALOG.get(order.item)
if not item or item[“qty”] < order.qty:
await ctx.send(sender, Receipt(item=order.item, qty=0, total=0, ok=False, note=”Not enough stock”))
return
total = item[“price”] * order.qty
item[“qty”] -= order.qty
await ctx.send(sender, Receipt(item=order.item, qty=order.qty, total=total, ok=True, note=”Thanks!”))

We create the Seller agent’s catalog and implement logic for responding to offer requests and processing orders. We simulate real-world trading by adding variable pricing and stock management, showing how the seller negotiates and completes transactions. Check out the Full Codes here.

Copy CodeCopiedUse a different Browser@buyer.on_event(“startup”)
async def buyer_start(ctx: Context):
ctx.logger.info(“Buyer querying directory for electronics…”)
resp = await ctx.ask(directory.address, ServiceQuery(category=”electronics”), expects=ServiceList, timeout=5.0)
sellers = resp.addresses if resp else []
if not sellers:
return
target = sellers[0]
desired = “laptop”
budget = 700
ctx.logger.info(f”Requesting offer for ‘{desired}’ within budget {budget} from {target}”)
offer = await ctx.ask(target, OfferRequest(item=desired, max_price=budget), expects=Offer, timeout=5.0)
if not offer or offer.price <= 0:
return
qty = 1 if offer.qty >= 1 else 0
if qty == 0:
return
ctx.logger.info(f”Placing order for {qty} x {offer.item} at {offer.price}”)
receipt = await ctx.ask(target, Order(item=offer.item, qty=qty), expects=Receipt, timeout=5.0)
if receipt and receipt.ok:
ctx.logger.info(f”ORDER SUCCESS: {receipt.qty} x {receipt.item} | total={receipt.total}”)

We program the Buyer agent to discover sellers, request offers, and place orders based on availability and budget. We observe how the buyer interacts with the seller through asynchronous communication to complete a purchase successfully. Check out the Full Codes here.

Copy CodeCopiedUse a different Browser@buyer.on_interval(period=6.0)
async def periodic_discovery(ctx: Context):
seen = await ctx.storage.get(“seen”) or 0
if seen >= 1:
return
await ctx.storage.set(“seen”, seen + 1)
ctx.logger.info(“Periodic discovery tick -> re-query directory”)
resp = await ctx.ask(directory.address, ServiceQuery(category=”electronics”), expects=ServiceList, timeout=3.0)
n = len(resp.addresses) if resp else 0
ctx.logger.info(f”Periodic: directory reports {n} seller(s)”)

bureau = Bureau()
bureau.add(directory)
bureau.add(seller)
bureau.add(buyer)

async def run_demo(seconds=10):
task = asyncio.create_task(bureau.run_async())
try:
await asyncio.sleep(seconds)
finally:
task.cancel()
try:
await task
except asyncio.CancelledError:
pass
print(“n Demo run complete.n”)

try:
loop = asyncio.get_running_loop()
await run_demo(10)
except RuntimeError:
asyncio.run(run_demo(10))

We add periodic discovery to have the buyer recheck available sellers, then have the Bureau run all agents together. We launch the asynchronous runtime to see the full marketplace simulation unfold and complete smoothly.

In conclusion, we have seen our agents discover one another, negotiate an offer, and complete a transaction entirely through message-based interactions. We realize how uAgents simplifies multi-agent orchestration by combining structure, communication, and state management seamlessly within Python. As we run this example, we not only witness a dynamic, autonomous system in action but also gain insight into how the same architecture can be extended to complex decentralized marketplaces, AI collaborations, and intelligent service networks, all within a lightweight, easy-to-use framework.

Check out the Full Codes here. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post A Coding Guide to Build a Fully Functional Multi-Agent Marketplace Using uAgent appeared first on MarkTechPost.

Graph databases have revolutionized how organizations manage complex, interconnected data. However, specialized query languages such as Gremlin often create a barrier for teams looking to extract insights efficiently. Unlike traditional relational databases with well-defined schemas, graph databases lack a centralized schema, requiring deep technical expertise for effective querying.
To address this challenge, we explore an approach that converts natural language to Gremlin queries, using Amazon Bedrock models such as Amazon Nova Pro. This approach helps business analysts, data scientists, and other non-technical users access and interact with graph databases seamlessly.
In this post, we outline our methodology for generating Gremlin queries from natural language, comparing different techniques and demonstrating how to evaluate the effectiveness of these generated queries using large language models (LLMs) as judges.
Solution overview
Transforming natural language queries into Gremlin queries requires a deep understanding of graph structures and the domain-specific knowledge encapsulated within the graph database. To achieve this, we divided our approach into three key steps:

Understanding and extracting graph knowledge
Structuring the graph similar to text-to-SQL processing
Generating and executing Gremlin queries

The following diagram illustrates this workflow.

Step 1: Extract graph knowledge
A successful query generation framework must integrate both graph knowledge and domain knowledge to accurately translate natural language queries. Graph knowledge encompasses structural and semantic information extracted directly from the graph database. Specifically, it includes:

Vertex labels and properties – A listing of vertex types, names, and their associated attributes
Edge labels and properties – Information about edge types and their attributes
One-hop neighbors for each vertex – Capturing local connectivity information, such as direct relationships between vertices

With this graph-specific knowledge, the framework can effectively reason about the heterogeneous properties and complex connections inherent to graph databases.
Domain knowledge captures additional context that augments the graph knowledge and is tailored specifically to the application domain. It is sourced in two ways:

Customer-provided domain knowledge – For example, the customer kscope.ai helped specify those vertices that represent metadata and should never be queried. Such constraints are encoded to guide the query generation process.
LLM-generated descriptions – To enhance the system’s understanding of vertex labels and their relevance to specific questions, we use an LLM to generate detailed semantic descriptions of vertex names, properties, and edges. These descriptions are stored within the domain knowledge repository and provide additional context to improve the relevance of the generated queries.

Step 2: Structure the graph as a text-to-SQL schema
To improve the model’s comprehension of graph structures, we adopt an approach similar to text-to-SQL processing, where we construct a schema representing vertex types, edges, and properties. This structured representation enhances the model’s ability to interpret and generate meaningful queries.
The question processing component transforms natural language input into structured elements for query generation. It operates in three stages:

Entity recognition and classification – Identifies key database elements in the input question (such as vertices, edges, and properties) and categorizes the question based on its intent
Context enhancement – Enriches the question with relevant information from the knowledge component, so both graph-specific and domain-specific context is properly captured
Query planning – Maps the enhanced question to specific database elements needed for query execution

The context generation component makes sure the generated queries accurately reflect the underlying graph structure by assembling the following:

Element properties – Retrieves attributes of vertices and edges along with their data types
Graph structure – Facilitates alignment with the database’s topology
Domain rules – Applies business constraints and logic

Step 3: Generate and execute Gremlin queries
The final step is query generation, where the LLM constructs a Gremlin query based on the extracted context. The process follows these steps:

The LLM generates an initial Gremlin query.
The query is executed within a Gremlin engine.
If the execution is successful, results are returned.
If execution fails, an error message parsing mechanism analyzes the returned errors and refines the query using LLM-based feedback.

This iterative refinement makes sure the generated queries align with the database’s structure and constraints, improving overall accuracy and usability.
Prompt template
Our final prompt template is as follows:

## Request
Please write a gremlin query to answer the given question:
{{question}}
You will be provided with couple relevant vertices, together with their
schema and other information.
Please choose the most relevant vertex according to its schema and other
information to make the gremlin query correct.

## Instructions
1. Here are related vertices and their details:
{{schema}}
2. Don’t rename properties.
3. Don’t change lines (using slash n) in the generated query.

## IMPORTANT
Return the results in the following XML format:

<Results>
<Query>INSERT YOUR QUERY HERE</Query>
<Explanation>
PROVIDE YOUR EXPLANATION ON HOW THIS QUERY WAS GENERATED
AND HOW THE PROVIDED SCHEMA WAS LEVERAGED
</Explanation>
</Results>

Comparing LLM-generated queries to ground truth
We implemented an LLM-based evaluation system using Anthropic’s Claude 3.5 Sonnet on Amazon Bedrock as a judge to assess both query generation and execution results for Amazon Nova Pro and a benchmark model. The system operates in two key areas:

Query evaluation – Assesses correctness, efficiency, and similarity to ground-truth queries; calculates exact matching component percentages; and provides an overall rating based on predefined rules developed with domain experts
Execution evaluation – Initially used a single-stage approach to compare generated results with ground truth, then enhanced to a two-stage evaluation process:

Item-by-item verification against ground truth
Calculation of overall match percentage

Testing across 120 questions demonstrated the framework’s ability to effectively distinguish correct from incorrect queries. The two-stage approach particularly improved the reliability of execution result evaluation by conducting thorough comparison before scoring.
Experiments and results
In this section, we discuss the experiments we conducted and their results.
Query similarity
In the query evaluation case, we propose two metrics: query exact match and query overall rating. An exact match score is calculated by identifying matching vs. non-matching components between generated and ground truth queries. The following table summarizes the scores for query exact match.

Easy
Medium
Hard
Overall

Amazon Nova Pro
82.70%
61%
46.60%
70.36%

Benchmark Model
92.60%
68.70%
56.20%
78.93%

An overall rating is provided after considering factors including query correctness, efficiency, and completeness as instructed in the prompt. The overall rating is on scale 1–10. The following table summarizes the scores for query overall rating.

Easy
Medium
Hard
Overall

Amazon Nova Pro
8.7
7
5.3
7.6

Benchmark Model
9.7
8
6.1
8.5

One limitation in the current query evaluation setup is that we rely solely on the LLM’s ability to compare ground truth against LLM-generated queries and arrive at the final scores. As a result, the LLM can fail to align with human preferences and under- or over-penalize the generated query. To address this, we recommend working with a subject matter expert to include domain-specific rules in the evaluation prompt.
Execution accuracy
To calculate accuracy, we compare the results of the LLM-generated Gremlin queries against the results of ground truth queries. If the results from both queries match exactly, we count the instance as correct; otherwise, it is considered incorrect. Accuracy is then computed as the ratio of correct query executions to the total number of queries tested. This metric provides a straightforward evaluation of how well the model-generated queries retrieve the expected information from the graph database, facilitating alignment with the intended query logic.
The following table summarizes the scores for execution results count match.

Easy
Medium
Hard
Overall

Amazon Nova Pro
80%
50%
10%
60.42%

Benchmark Model
90%
70%
30%
74.83%

Query execution latency
In addition to accuracy, we evaluate the efficiency of generated queries by measuring their runtime and comparing it with the ground truth queries. For each query, we record the runtime in milliseconds and analyze the difference between the generated query and the corresponding ground truth query. A lower runtime indicates a more optimized query, whereas significant deviations might suggest inefficiencies in query structure or execution planning. By considering both accuracy and runtime, we gain a more comprehensive assessment of query quality, making sure the generated queries are correct and performant within the graph database. The following box plot showcases query execution latency with respect to time for the ground truth query and the query generated by Amazon Nova Pro. As illustrated, all three types of queries exhibit comparable runtimes, with similar median latencies and overlapping interquartile ranges. Although the ground truth queries display a slightly wider range and a higher outlier, the median values across all three groups remain close. This suggests that the model-generated queries are at the same level as human-written ones in terms of execution efficiency, supporting the claim that AI-generated queries are of similar quality and don’t incur additional latency overhead.

Query generation latency and cost
Finally, we compare the time taken to generate each query and calculate the cost based on token consumption. More specifically, we measure the query generation time and track the number of tokens used, because most LLM-based APIs charge based on token usage. By analyzing both the generation speed and token cost, we can determine whether the model is efficient and cost-effective. These results provide insights in selecting the optimal model that balances query accuracy, execution efficiency, and economic feasibility.
As shown in the following plots, Amazon Nova Pro consistently outperforms the benchmark model in both generation latency and cost. In the left plot, which depicts query generation latency, Amazon Nova Pro demonstrates a significantly lower median generation time, with most values clustered between 1.8–4 seconds, compared to the benchmark model’s broader range from around 5–11 seconds. The right plot, illustrating query generation cost, shows that Amazon Nova Pro maintains a much smaller cost per query—centered well below $0.005—whereas the benchmark model incurs higher and more variable costs, reaching up to $0.025 in some cases. These results highlight Amazon Nova Pro’s advantage in terms of both speed and affordability, making it a strong candidate for deployment in time-sensitive or large-scale systems.

Conclusion
We experimented with all 120 ground truth queries provided to us by kscope.ai and achieved an overall accuracy of 74.17% in generating correct results. The proposed framework demonstrates its potential by effectively addressing the unique challenges of graph query generation, including handling heterogeneous vertex and edge properties, reasoning over complex graph structures, and incorporating domain knowledge. Key components of the framework, such as the integration of graph and domain knowledge, the use of Retrieval Augmented Generation (RAG) for query plan creation, and the iterative error-handling mechanism for query refinement, have been instrumental in achieving this performance.
In addition to improving accuracy, we are actively working on several enhancements. These include refining the evaluation methodology to handle deeply nested query results more effectively and further optimizing the use of LLMs for query generation. Moreover, we are using the RAGAS-faithfulness metric to improve the automated evaluation of query results, resulting in greater reliability and consistency in assessing the framework’s outputs.

About the authors
Mengdie (Flora) Wang is a Data Scientist at AWS Generative AI Innovation Center, where she works with customers to architect and implement scalable Generative AI solutions that address their unique business challenges. She specializes in model customization techniques and agent-based AI systems, helping organizations harness the full potential of generative AI technology. Prior to AWS, Flora earned her Master’s degree in Computer Science from the University of Minnesota, where she developed her expertise in machine learning and artificial intelligence.
Jason Zhang has expertise in machine learning, reinforcement learning, and generative AI. He earned his Ph.D. in Mechanical Engineering in 2014, where his research focused on applying reinforcement learning to real-time optimal control problems. He began his career at Tesla, applying machine learning to vehicle diagnostics, then advanced NLP research at Apple and Amazon Alexa. At AWS, he worked as a Senior Data Scientist on generative AI solutions for customers.
Rachel Hanspal is a Deep Learning Architect at AWS Generative AI Innovation Center, specializing in end-to-end GenAI solutions with a focus on frontend architecture and LLM integration. She excels in translating complex business requirements into innovative applications, leveraging expertise in natural language processing, automated visualization, and secure cloud architectures.
Zubair Nabi is the CTO and Co-Founder of Kscope, an Integrated Security Posture Management (ISPM) platform. His expertise lies at the intersection of Big Data, Machine Learning, and Distributed Systems, with over a decade of experience building software, data, and AI platforms. Zubair is also an adjunct faculty member at George Washington University and the author of Pro Spark Streaming: The Zen of Real-Time Analytics Using Apache Spark. He holds an MPhil from the University of Cambridge.
Suparna Pal – CEO & Co-Founder of kscope.ai – 20+ years of journey of building innovative platforms & solutions for Industrial, Health Care and IT operations at PTC, GE, and Cisco.
Wan Chen is an Applied Science Manager at AWS Generative AI Innovation Center. As a ML/AI veteran in tech industry, she has wide range of expertise on traditional machine learning, recommender system, deep learning and Generative AI. She is a stronger believer of Superintelligence and is very passionate to push the boundary of AI research and application to enhance human life and drive business growth. She holds Ph.D in Applied Mathematics from University of British Columbia and had worked as postdoctoral fellow in Oxford University.
Mu Li is a Principal Solutions Architect with AWS Energy. He’s also the Worldwide Tech Leader for the AWS Energy & Utilities Technical Field Community (TFC), a community of 300+ industry and technical experts. Li is passionate about working with customers to achieve business outcomes using technology. Li has worked with customers to migrate all-in to AWS from on-prem and Azure, launch the Production Monitoring and Surveillance industry solution, deploy ION/OpenLink Endur on AWS, and implement AWS-based IoT and machine learning workloads. Outside of work, Li enjoys spending time with his family, investing, following Houston sports teams, and catching up on business and technology.

Over the past two years, companies have seen an increasing need to develop a project prioritization methodology for generative AI. There is no shortage of generative AI use cases to consider. Rather, companies want to evaluate the business value against the cost, level of effort, and other concerns, for a large number of potential generative AI projects. One new concern for generative AI compared to other domains is considering issues like hallucination, generative AI agents making incorrect decisions and then acting on those decisions through tool calls to downstream systems, and dealing with the rapidly changing regulatory landscape. In this post we describe how to incorporate responsible AI practices into a prioritization method to systematically address these types of concerns.
Responsible AI overview
The AWS Well-Architected Framework defines responsible AI as “the practice of designing, developing, and using AI technology with the goal of maximizing benefits and minimizing risks.” The AWS responsible AI framework begins by defining eight dimensions of responsible AI: fairness, explainability, privacy and security, safety, controllability, veracity and robustness, governance, and transparency. At key points in the development lifecycle, a generative AI team should consider the possible harms or risks for each dimension (inherent and residual risks), implements risk mitigations, and monitors risk on an ongoing basis. Responsible AI applies across the entire development lifecycle and should be considered during initial project prioritization. That’s especially true for generative AI projects, where there are novel types of risks to consider, and mitigations might not be as well understood or researched. Considering responsible AI up front gives a more accurate picture of project risk and mitigation level of effort and reduces the chance of costly rework if risks are uncovered later in the development lifecycle. In addition to potentially delayed projects due to rework, unmitigated concerns might also harm customer trust, result in representational harm, or fail to meet regulatory requirements.
Generative AI prioritization
While most companies have their own prioritization methods, here we’ll demonstrate how to use the weighted shortest job first (WSJF) method from the Scaled Agile system. WSJF assigns a priority using this formula:
Priority = (cost of delay) / (job size)
The cost of delay is a measure of business value. It includes the direct value (for example, additional revenue or cost savings), the timeliness (such as, is shipping this project worth a lot more today than a year from now), and the adjacent opportunities (such as, would delivering this project open up other opportunities down the road).
The job size is where you consider the level of effort to deliver the project. That normally includes direct development costs and paying for any infrastructure or software you need. The job size is where you can include the results of the initial responsible AI risk assessment and expected mitigations. For example, if the initial assessment uncovers three risks that require mitigation, you include the development cost for those mitigations in the job size. You can also qualitatively assess that a project with ten high-priority risks is more complex than a project with only two high-priority risks.
Example scenario
Now, let’s walk through a prioritization exercise that compares two generative AI projects. The first project uses a large language model (LLM) to generate product descriptions. A marketing team will use this application to automatically create production descriptions that go into the online product catalog website. The second project uses a text-to-image model to generate new visuals for advertising campaigns and the product catalog. The marketing team will use this application to more quickly create customized brand assets.
First pass prioritization
First, we’ll go through the prioritization method without considering responsible AI, assigning a score of 1–5 for each part of the WSJF formula. The specific scores vary by organization. Some companies prefer to use t-shirt sizing (S, M, L, and XL), others prefer a score of 1–5, and others will use a more granular score. A score of 1–5 is a common and straightforward way to start. For example, the direct value scores can be calculated as:
1 = no direct value
2 = 20% improvement in KPI (time to create high-quality descriptions)
3 = 40% improvement in KPI
4 = 80% improvement in KPI
5 = 100% or more improvement in KPI

Project 1: Automated product descriptions (scored from 1–5)
Project 2: Creating visual brand assets (scored from 1–5)

Direct value
3: Helps marketing team create higher quality descriptions more quickly
3: Helps marketing team create higher quality assets more quickly

Timeliness
2: Not particularly urgent
4: New ad campaign planned this quarter; without this project, cannot create enough brand assets without hiring a new agency to supplement the team

Adjacent opportunities
2: Might be able to reuse for similar scenarios)
3: Experience gained in image generation will build competence for future projects

Job size
2: Basic, well-known pattern
2: Basic, well-known pattern

Score
(3+2+2)/2 = 3.5
(3+4+3)/2 = 5

At first glance, it looks like Project 2 is more compelling. Intuitively that makes sense—it takes people a lot longer to make high-quality visuals than to create textual product descriptions.
Risk assessment
Now let’s go through a risk assessment for each project. The following table lists a brief overview of the outcome of a risk assessment along each of the AWS responsible AI dimensions, along with a t-shirt size (S, M, L, and XL) severity level. The table also includes suggested mitigations.

Project 1: Automated product descriptions
Project 2: Creating visual brand assets

Fairness
L: Are descriptions appropriate in terms of gender and demographics? Mitigate using guardrails.
L: Images must not portray particular demographics in a biased way. Mitigate using human and automated checks.

Explainability
No risks identified.
No risks identified.

Privacy and security
L: Some product information is proprietary and cannot be listed on a public site. Mitigate using data governance.
L: Model must not be trained on any images that contain proprietary information. Mitigate using data governance.

Safety
M: Language must be age-appropriate and not cover offensive topics. Mitigate using guardrails.
L: Images must not contain adult content or images of drugs, alcohol, or weapons. Mitigate using guardrails.

Controllability
S: Need to track customer feedback on the descriptions. Mitigate using customer feedback collection.
L: Do images align to our brand guidelines? Mitigate using human and automated checks.

Veracity and robustness
M: Will the system hallucinate and imply product capabilities that aren’t real? Mitigate using guardrails.
L: Are images realistic enough to avoid uncanny valley effects? Mitigate using human and automated checks.

Governance
M: Prefer LLM providers that offer copyright indemnification. Mitigate using LLM provider selection.
L: Require copyright indemnification and image source attribution. Mitigate using model provider selection.

Transparency
S: Disclose that descriptions are AI generated.
S: Disclose that descriptions are AI generated.

The risks and mitigations are use-case specific. The preceding table is for illustrative purposes only.
Second pass prioritization
How does the risk assessment affect the prioritization?

Project 1: Automated product descriptions (scored from 1–5)
Project 2: Creating visual brand assets (scored from 1–5)

Job size
3: Basic, well-known pattern; requires fairly standard guardrails, governance, and feedback collection.
5: Basic, well-known pattern. Requires advanced image guardrails with human oversight, and a more expensive commercial model. Research spike needed.

Score
(3+2+2)/3 = 2.3
(3+4+3)/5 = 2

Now it looks like Project 1 is a better one to start with. Intuitively, after you consider responsible AI, that makes sense. Poorly crafted or offensive images are more noticeable and have a larger impact than a poorly phrased product description. And the guardrails you can use for maintaining image safety are less mature than the equivalent guardrails for text, particularly in ambiguous cases like adhering to brand guidelines. In fact, an image guardrail system might require training a monitoring model or using people to spot-check some percentage of the output. You might need to dedicate a small science team to study this problem first.
Conclusion
In this post, you saw how to include responsible AI considerations in a generative AI project prioritization method. You saw how conducting a responsible AI risk assessment in the initial prioritization phase can change the outcome by uncovering a substantial amount of mitigation work. Moving forward, you should develop your own responsible AI policy and start adopting responsible AI practices for generative AI projects. You can find additional details and resources at Transform responsible AI from theory into practice.

About the author
Randy DeFauw is a Sr. Principal Solutions Architect at AWS. He has over 20 years of experience in technology, starting with his university work on autonomous vehicles. He has worked with and for customers ranging from startups to Fortune 50 companies, launching Big Data and Machine Learning applications. He holds an MSEE and an MBA, serves as a board advisor to K-12 STEM education initiatives, and has spoken at leading conferences including Strata and GlueCon. He is the co-author of the books SageMaker Best Practices and Generative AI Cloud Solutions. Randy currently acts as a technical advisor to AWS’ director of technology in North America.

In this tutorial, we explore how we can build a compact yet powerful Enterprise AI assistant that runs effortlessly on Colab. We start by integrating retrieval-augmented generation (RAG) using FAISS for document retrieval and FLAN-T5 for text generation, both fully open-source and free. As we progress, we embed enterprise policies such as data redaction, access control, and PII protection directly into the workflow, ensuring our system is intelligent and compliant. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browser!pip -q install faiss-cpu transformers==4.44.2 accelerate sentence-transformers==3.0.1

from typing import List, Dict, Tuple
import re, textwrap, numpy as np, torch
from sentence_transformers import SentenceTransformer
import faiss
from transformers import pipeline, AutoTokenizer, AutoModelForSeq2SeqLM

GEN_MODEL = “google/flan-t5-base”
EMB_MODEL = “sentence-transformers/all-MiniLM-L6-v2″

gen_tok = AutoTokenizer.from_pretrained(GEN_MODEL)
gen_model = AutoModelForSeq2SeqLM.from_pretrained(GEN_MODEL, device_map=”auto”)
generate = pipeline(“text2text-generation”, model=gen_model, tokenizer=gen_tok)

emb_device = “cuda” if torch.cuda.is_available() else “cpu”
emb_model = SentenceTransformer(EMB_MODEL, device=emb_device)

We begin by setting up our environment and loading the required models. We initialize FLAN-T5 for text generation and MiniLM for embedding representations. We ensure both models are configured to automatically use the GPU when available, so our pipeline runs efficiently. Check out the FULL CODES here.

Copy CodeCopiedUse a different BrowserDOCS = [
{“id”:”policy_sec_001″,”title”:”Data Security Policy”,
“text”:”All customer data must be encrypted at rest (AES-256) and in transit (TLS 1.2+). Access is role-based (RBAC). Secrets are stored in a managed vault. Backups run nightly with 35-day retention. PII includes name, email, phone, address, PAN/Aadhaar.”},
{“id”:”policy_ai_002″,”title”:”Responsible AI Guidelines”,
“text”:”Use internal models for confidential data. Retrieval sources must be logged. No customer decisioning without human-in-the-loop. Redact PII in prompts and outputs. All model prompts and outputs are stored for audit for 180 days.”},
{“id”:”runbook_inc_003″,”title”:”Incident Response Runbook”,
“text”:”If a suspected breach occurs, page on-call SecOps. Rotate keys, isolate affected services, perform forensic capture, notify DPO within regulatory SLA. Communicate via the incident room only.”},
{“id”:”sop_sales_004″,”title”:”Sales SOP – Enterprise Deals”,
“text”:”For RFPs, use the approved security questionnaire responses. Claims must match policy_sec_001. Custom clauses need Legal sign-off. Keep records in CRM with deal room links.”}
]

def chunk(text:str, chunk_size=600, overlap=80):
w = text.split()
if len(w) <= chunk_size: return [text]
out=[]; i=0
while i < len(w):
j=min(i+chunk_size, len(w)); out.append(” “.join(w[i:j]))
if j==len(w): break
i = j – overlap
return out

CORPUS=[]
for d in DOCS:
for i,c in enumerate(chunk(d[“text”])):
CORPUS.append({“doc_id”:d[“id”],”title”:d[“title”],”chunk_id”:i,”text”:c})

We create a small enterprise-style document set to simulate internal policies and procedures. We then break these long texts into manageable chunks so they can be embedded and retrieved effectively. This chunking helps our AI assistant handle contextual information with better precision. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserdef build_index(chunks:List[Dict]) -> Tuple[faiss.IndexFlatIP, np.ndarray]:
vecs = emb_model.encode([c[“text”] for c in chunks], normalize_embeddings=True, convert_to_numpy=True)
index = faiss.IndexFlatIP(vecs.shape[1]); index.add(vecs); return index, vecs

INDEX, VECS = build_index(CORPUS)

PII_PATTERNS = [
(re.compile(r”bd{10}b”), “<REDACTED_PHONE>”),
(re.compile(r”b[A-Z0-9._%+-]+@[A-Z0-9.-]+.[A-Z]{2,}b”, re.I), “<REDACTED_EMAIL>”),
(re.compile(r”bd{12}b”), “<REDACTED_ID12>”),
(re.compile(r”b[A-Z]{5}d{4}[A-Z]b”), “<REDACTED_PAN>”)
]
def redact(t:str)->str:
for p,r in PII_PATTERNS: t = p.sub(r, t)
return t

POLICY_DISALLOWED = [
re.compile(r”b(share|exfiltrate)b.*b(raw|all)b.*bdatab”, re.I),
re.compile(r”bdisableb.*bencryptionb”, re.I),
]
def policy_check(q:str):
for r in POLICY_DISALLOWED:
if r.search(q): return False, “Request violates security policy (data exfiltration/encryption tampering).”
return True, “”

We embed all chunks using Sentence Transformers and store them in a FAISS index for fast retrieval. We introduce PII redaction rules and policy checks to prevent misuse of data. By doing this, we ensure our assistant adheres to enterprise security and compliance guidelines. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserdef retrieve(query:str, k=4)->List[Dict]:
qv = emb_model.encode([query], normalize_embeddings=True, convert_to_numpy=True)
scores, idxs = INDEX.search(qv, k)
return [{**CORPUS[i], “score”: float(s)} for s,i in zip(scores[0], idxs[0])]

SYSTEM = (“You are an enterprise AI assistant.n”
“- Answer strictly from the provided CONTEXT.n”
“- If missing info, say what is unknown and suggest the correct policy/runbook.n”
“- Keep it concise and cite titles + doc_ids inline like [Title (doc_id:chunk)].”)
def build_prompt(user_q:str, ctx_blocks:List[Dict])->str:
ctx = “nn”.join(f”[{i+1}] {b[‘title’]} (doc:{b[‘doc_id’]}:{b[‘chunk_id’]})n{b[‘text’]}” for i,b in enumerate(ctx_blocks))
uq = redact(user_q)
return f”SYSTEM:n{SYSTEM}nnCONTEXT:n{ctx}nnUSER QUESTION:n{uq}nnINSTRUCTIONS:n- Cite sources inline.n- Keep to 5-8 sentences.n- Preserve redactions.”

def answer(user_q:str, k=4, max_new_tokens=220)->Dict:
ok,msg = policy_check(user_q)
if not ok: return {“answer”: f” {msg}”, “ctx”:[]}
ctx = retrieve(user_q, k=k); prompt = build_prompt(user_q, ctx)
out = generate(prompt, max_new_tokens=max_new_tokens, do_sample=False)[0][“generated_text”].strip()
return {“answer”: out, “ctx”: ctx}

We design the retrieval function to fetch relevant document sections for each user query. We then construct a structured prompt combining context and questions for FLAN-T5 to generate precise answers. This step ensures that our assistant produces grounded, policy-compliant responses. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserdef eval_query(user_q:str, ctx:List[Dict])->Dict:
terms = [w.lower() for w in re.findall(r”[a-zA-Z]{4,}”, user_q)]
ctx_text = ” “.join(c[“text”].lower() for c in ctx)
hits = sum(t in ctx_text for t in terms)
return {“terms”: len(terms), “hits”: hits, “hit_rate”: round(hits/max(1,len(terms)), 2)}

QUERIES = [
“What encryption and backup rules do we follow for customer data?”,
“Can we auto-answer RFP security questionnaires? What should we cite?”,
“If there is a suspected breach, what are the first three steps?”,
“Is it allowed to share all raw customer data externally for testing?”
]
for q in QUERIES:
res = answer(q, k=3)
print(“n” + “=”*100); print(“Q:”, q); print(“nA:”, res[“answer”])
if res[“ctx”]:
ev = eval_query(q, res[“ctx”]); print(“nRetrieved Context (top 3):”)
for r in res[“ctx”]: print(f”- {r[‘title’]} [{r[‘doc_id’]}:{r[‘chunk_id’]}] score={r[‘score’]:.3f}”)
print(“Eval:”, ev)

We evaluate our system using sample enterprise queries that test encryption, RFPs, and incident procedures. We display retrieved documents, answers, and simple hit-rate scores to check relevance. Through this demo, we observe our Enterprise AI assistant performing retrieval-augmented reasoning securely and accurately.

In conclusion, we successfully created a self-contained enterprise AI system that retrieves, analyzes, and responds to business queries while maintaining strong guardrails. We appreciate how seamlessly we can combine FAISS for retrieval, Sentence Transformers for embeddings, and FLAN-T5 for generation to simulate an internal enterprise knowledge engine. As we finish, we realize that this simple Colab-based implementation can serve as a blueprint for scalable, auditable, and compliant enterprise deployments.

Check out the FULL CODES here. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post How to Design a Fully Functional Enterprise AI Assistant with Retrieval Augmentation and Policy Guardrails Using Open Source AI Models appeared first on MarkTechPost.

Pokee AI has open sourced PokeeResearch-7B, a 7B parameter deep research agent that executes full research loops, decomposes a query, issues search and read calls, verifies candidate answers, then synthesizes multiple research threads into a final response.

The agent runs a research and verification loop. In research, it calls external tools for web search and page reading or proposes an interim answer. In verification, it checks the answer against retrieved evidence, and either accepts or restarts research. This structure reduces brittle trajectories and catches obvious errors before finalization. The research team formalizes this loop and adds a test-time synthesis stage that merges several independent research threads.

Training recipe, RLAIF with RLOO

PokeeResearch-7B is finetuned from Qwen2.5-7B-Instruct using an annotation-free Reinforcement Learning from AI Feedback, called RLAIF, with the REINFORCE Leave-One-Out algorithm, called RLOO. The reward targets semantic correctness, citation faithfulness, and instruction adherence, not token overlap. The Model’s Hugging Face card lists batch size 64, 8 research threads per prompt during RL, learning rate 3e-6, 140 steps, context 32,768 tokens, bf16 precision, and a checkpoint near 13 GB. The research team emphasizes that RLOO provides an unbiased on policy gradient and contrasts it with the PPO family that is approximately on policy and biased.

https://arxiv.org/pdf/2510.15862

Reasoning scaffold and Research Threads Synthesis

The scaffold includes three mechanisms. Self correction, the agent detects malformed tool calls and retries. Self verification, the agent inspects its own answer against evidence. Research Threads Synthesis, the agent runs several independent threads per question, summarizes them, then synthesizes a final answer. The research team reports that synthesis improves accuracy on difficult benchmarks.

https://arxiv.org/pdf/2510.15862

Evaluation protocol

The research team evaluates text only questions from 10 benchmarks, NQ, TriviaQA, PopQA, HotpotQA, 2WikiMultiHopQA, Musique, Bamboogle, GAIA, BrowseComp, and Humanity’s Last Exam. They sample 125 questions per dataset, except GAIA with 103, for a total of 1,228 questions. For each question, they run 4 research threads, then compute mean accuracy, mean at 4, using Gemini-2.5-Flash-lite to judge correctness. The maximum interaction turns are set to 100.

https://github.com/Pokee-AI/PokeeResearchOSS

https://github.com/Pokee-AI/PokeeResearchOSS

Results at 7B scale

PokeeResearch-7B reports the best mean at 4 accuracy among 7B deep research agents across the 10 datasets. On HLE the model reports 15.2 without RTS and 17.6 with RTS. On GAIA the model reports 36.9 without RTS and 41.3 with RTS. On BrowseComp the model reports 5.4 without RTS and 8.4 with RTS. On the seven QA benchmarks, Bamboogle, 2WikiMultiHopQA, TriviaQA, NQ, PopQA, Musique, HotpotQA, the model improves over recent 7B baselines. Gains from RTS are largest on HLE, GAIA, and BrowseComp, and smaller on the QA sets.

Key Takeaways

Training: PokeeResearch-7B fine tunes Qwen2.5-7B-Instruct with RLAIF using the RLOO estimator, optimizing rewards for factual accuracy, citation faithfulness, and instruction adherence, not token overlap.

Scaffold: The agent runs a research and verification loop with Research Threads Synthesis, executing multiple independent threads, then synthesizing evidence to a final answer.

Evaluation protocol: Benchmarks span 10 datasets with 125 questions each, except GAIA with 103, 4 threads per question, mean@4 accuracy judged by Gemini-2.5-Flash-lite, with a 100 turn cap.

Results and release: PokeeResearch-7B reports state of the art among 7B deep research agents, for example HLE 17.6 with RTS, GAIA 41.3 with RTS, BrowseComp 8.4 with RTS, and is released under Apache-2.0 with code and weights public.

Editorial Comments

PokeeResearch-7B is a useful step for practical deep research agents. It aligns training with RLAIF using RLOO, so the objective targets semantic correctness, citation faithfulness, and instruction adherence. The reasoning scaffold includes self verification and Research Threads Synthesis, which improves difficult benchmarks. The evaluation uses mean at 4 with Gemini 2.5 Flash lite as the judge, across 10 datasets. The release ships Apache 2.0 code and weights with a clear tool stack using Serper and Jina. The setup runs on a single A100 80 GB and scales.

Check out the Paper, Model on HF and GitHub Repo. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post PokeeResearch-7B: An Open 7B Deep-Research Agent Trained with Reinforcement Learning from AI Feedback (RLAIF) and a Robust Reasoning Scaffold appeared first on MarkTechPost.

TLDR: VISTA is a multi agent framework that improves text to video generation during inference, it plans structured prompts as scenes, runs a pairwise tournament to select the best candidate, uses specialized judges across visual, audio, and context, then rewrites the prompt with a Deep Thinking Prompting Agent, the method shows consistent gains over strong prompt optimization baselines in single scene and multi scene settings, and human raters prefer its outputs.

https://arxiv.org/pdf/2510.15831

What VISTA is?

VISTA stands for Video Iterative Self improvemenT Agent. It is a black box, multi agent loop that refines prompts and regenerates videos at test time. The system targets 3 aspects jointly, visual, audio, and context. It follows 4 steps, structured video prompt planning, pairwise tournament selection, multi dimensional multi agent critiques, and a Deep Thinking Prompting Agent for prompt rewriting.

The research team evaluates VISTA on a single scene benchmark and on an internal multi scene set. It reports consistent improvements and up to 60 percent pairwise win rate against state of the art baselines in some settings, and a 66.4 percent human preference over the strongest baseline.

https://arxiv.org/pdf/2510.15831

Understanding the key problem

Text to video models like Veo 3 can produce high quality video and audio, yet outputs remain sensitive to exact prompt phrasing, adherence to physics can fail, and alignment to user goals can drift, which forces manual trial and error. VISTA frames this as a test time optimization problem. It seeks unified improvement across visual signals, audio signals, and contextual alignment.

How VISTA works, step by step?

Step 1: structured video prompt planning

The user prompt is decomposed into timed scenes. Each scene carries 9 properties, duration, scene type, characters, actions, dialogues, visual environment, camera, sounds, moods. A multimodal LLM fills missing properties and enforces constraints on realism, relevancy, and creativity by default. The system also keeps the original user prompt in the candidate set to allow models that do not benefit from decomposition.

Step 2: pairwise tournament video selection

The system samples multiple video, prompt pairs. An MLLM acts as a judge with binary tournaments and bidirectional swapping to reduce token order bias. The default criteria include visual fidelity, physical commonsense, text video alignment, audio video alignment, and engagement. The method first elicits probing critiques to support analysis, then performs pairwise comparison, and applies customizable penalties for common text to video failures.

Step 3: multi dimensional multi agent critiques

The champion video and prompt receive critiques along 3 dimensions, visual, audio, and context. Each dimension uses a triad, a normal judge, an adversarial judge, and a meta judge that consolidates both sides. Metrics include visual fidelity, motions and dynamics, temporal consistency, camera focus, and visual safety for visual, audio fidelity, audio video alignment, and audio safety for audio, situational appropriateness, semantic coherence, text video alignment, physical commonsense, engagement, and video format for context. Scores are on a 1 to 10 scale, which supports targeted error discovery.

Step 4: Deep Thinking Prompting Agent

The reasoning module reads the meta critiques and runs a 6 step introspection, it identifies low scoring metrics, clarifies expected outcomes, checks prompt sufficiency, separates model limits from prompt issues, detects conflicts or vagueness, proposes modification actions, then samples refined prompts for the next generation cycle.

https://arxiv.org/pdf/2510.15831

Understanding the results

Automatic evaluation: The research study reports win, tie, loss rates on ten criteria using an MLLM as a judge, with bidirectional comparisons. VISTA achieves a win rate over direct prompting that rises across iterations, reaching 45.9 percent in single scene and 46.3 percent in multi scene at iteration 5. It also wins directly against each baseline under the same compute budget.

Human studies: Annotators with prompt optimization experience prefer VISTA in 66.4 percent of head to head trials against the best baseline at iteration 5. Experts rate optimization trajectories higher for VISTA, and they score visual quality and audio quality higher than direct prompting.

Cost and scaling: Average tokens per iteration are about 0.7 million across two datasets, generation tokens are not included. Most token use comes from selection and critiques, which process videos as long context inputs. Win rate tends to increase as the number of sampled videos and tokens per iteration increases.

Ablations: Removing prompt planning weakens initialization. Removing tournament selection destabilizes later iterations. Using only one judge type reduces performance. Removing the Deep Thinking Prompting Agent lowers final win rates.

Evaluators: The research team repeated evaluation with alternative evaluator models and observe similar iterative improvements, which supports robustness of the trend.

https://arxiv.org/pdf/2510.15831

https://arxiv.org/pdf/2510.15831

Key Takeaways

VISTA is a test time, multi agent loop that jointly optimizes visual, audio, and context for text to video generation.

It plans prompts as timed scenes with 9 attributes, duration, scene type, characters, actions, dialogues, visual environment, camera, sounds, moods.

Candidate videos are selected via pairwise tournaments using an MLLM judge with bidirectional swap, scored on visual fidelity, physical commonsense, text video alignment, audio video alignment, and engagement.

A triad of judges per dimension, normal, adversarial, meta, produces 1 to 10 scores that guide the Deep Thinking Prompting Agent to rewrite the prompt and iterate.

Results show 45.9 percent wins on single scene and 46.3 percent on multi scene at iteration 5 over direct prompting, human raters prefer VISTA in 66.4 percent of trials, average token cost per iteration is about 0.7 million.

Editorial Comments

VISTA is a practical step toward reliable text to video generation, it treats inference as an optimization loop and keeps the generator as a black box. The structured video prompt planning is useful for early engineers, the 9 scene attributes give a concrete checklist. The pairwise tournament selection with a multimodal LLM judge and bidirectional swap is a sensible way to reduce ordering bias, the criteria target real failure modes, visual fidelity, physical commonsense, text video alignment, audio video alignment, engagement. The multi dimensional critiques separate visual, audio, and context, the normal, adversarial, and meta judges expose weaknesses that single judges miss. The Deep Thinking Prompting Agent turns those diagnostics into targeted prompt edits. The use of Gemini 2.5 Flash and Veo 3 clarifies the reference setup, the Veo 2 study is a helpful lower bound. The reported 45.9 and 46.3 percent win rates and 66.4 percent human preference indicate repeatable gains. The 0.7 million token cost is non trivial, yet transparent and scalable.

Check out the Paper and Project Page. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post Google AI Introduces VISTA: A Test Time Self Improving Agent for Text to Video Generation appeared first on MarkTechPost.

In Part 1 of our series, we introduced a proactive cost management solution for Amazon Bedrock, featuring a robust cost sentry mechanism designed to enforce real-time token usage limits. We explored the core architecture, token tracking strategies, and initial budget enforcement techniques that help organizations control their generative AI expenses.
Building upon that foundation, this post explores advanced cost monitoring strategies for generative AI deployments. We introduce granular custom tagging approaches for precise cost allocation, and develop comprehensive reporting mechanisms.
Solution overview
The cost sentry solution introduced in Part 1 was developed as a centralized mechanism to proactively limit generative AI usage to adhere to prescribed budgets. The following diagram illustrates the core components of the solution, adding in cost monitoring through AWS Billing and Cost Management.

Invocation-level tagging for enhanced traceability
Invocation-level tagging extends our solution’s capabilities by attaching rich metadata to every API request, creating a comprehensive audit trail within Amazon CloudWatch logs. This becomes particularly valuable when investigating budget-related decisions, analyzing rate-limiting impacts, or understanding usage patterns across different applications and teams. To support this, the main AWS Step Functions workflow was updated, as illustrated in the following figure.

Enhanced API input
We also evolved the API input to support custom tagging. The new input structure introduces optional parameters for model-specific configurations and custom tagging:

{
  “model”: “string”,     // e.g., “claude-3” or “anthropic.claude-3-sonnet-20240229-v1:0”
  “prompt”: {
    “messages”: [
      {
        “role”: “string”,    // “system”, “user”, or “assistant”
        “content”: “string”
      }
    ],
    “parameters”: {
      “max_tokens”: number,    // Optional, model-specific defaults
      “temperature”: number,   // Optional, model-specific defaults
      “top_p”: number,         // Optional, model-specific defaults
      “top_k”: number          // Optional, model-specific defaults
    }
  },
  “tags”: {
    “applicationId”: “string”,  // Required
    “costCenter”: “string”,     // Optional
    “environment”: “string”     // Optional – dev/staging/prod
  }
}

The input structure comprises three key components:

model – Maps simple names (for example, claude-3) to full Amazon Bedrock model IDs (for example, anthropic.claude-3-sonnet-20240229-v1:0)
input – Provides a messages array for prompts, supporting both single-turn and multi-turn conversations
tags – Supports application-level tracking, with applicationId as the required field and costCenter and environment as optional fields

In this example, we use different cost centers for sales, services, and support to simulate the use of a business attribute to track usage and spend for inference in Amazon Bedrock. For example:

{
  “model”: “claude-3-5-haiku”,
  “prompt”: {
    “messages”: [
      {
        “role”: “user”,
        “content”: “Explain the benefits of using S3 using only 100 words.”
      },
      {
        “role”: “assistant”,
        “content”: “You are a helpful AWS expert.”
      }
    ],
    “parameters”: {
      “max_tokens”: 2000,
      “temperature”: 0.7,
      “top_p”: 0.9,
      “top_k”: 50
    }
  },
  “tags”: {
    “applicationId”: “aws-documentation-helper”,
    “costCenter”: “support”,
    “environment”: “production”
  }
}

Validation and tagging
A new validation step was added to the workflow for tagging. This step uses an AWS Lambda function to add validation checks and maps the model requested to the specific model ID in Amazon Bedrock. It supplements the tags object with tags that will be required for downstream analysis.
The following code is an example of a simple map to get the appropriate model ID from the model specified:

MODEL_ID_MAPPING = {
    “nova-lite”: “amazon.nova-lite-v1:0”,
    “nova-micro”: “amazon.nova-micro-v1:0”,
    “claude-2”: “anthropic.claude-v2:0”,
    “claude-3-haiku”: “anthropic.claude-3-haiku-20240307-v1:0”,
    “claude-3-5-sonnet-v2”: “us.anthropic.claude-3-5-sonnet-20241022-v2:0”,
    “claude-3-5-haiku”: “us.anthropic.claude-3-5-haiku-20241022-v1:0”
}

Logging and analysis
By using CloudWatch metrics with custom-generated tags and dimensions, you can track detailed metrics across multiple dimensions such as model type, cost center, application, and environment. Custom tags and dimensions show how teams use AI services. To see this analysis, steps were implemented to generate custom tags, store metric data, and analyze metric data:

We include a unique set of tags that capture contextual information. This can include user-supplied tags as well as ones that are dynamically generated, such as requestId and timestamp:

  “tags”: {
    “requestId”: “ded98994-eb76-48d9-9dbc-f269541b5e49”,
    “timestamp”: “2025-01-31T14:05:26.854682”,
    “applicationId”: “aws-documentation-helper”,
    “costCenter”: “support”,
    “environment”: “production”
}

As each workflow is executed, the limit for each model will be evaluated to make sure the request is within budgetary guidelines. The workflow will end based on three possible outcomes:

Rate limit approved and invocation successful
Rate limit approved and invocation unsuccessful
Rate limit denied
The custom metric data is saved in CloudWatch in the GenAIRateLimiting namespace. This namespace includes the following key metrics:

TotalRequests – Counts every invocation attempt regardless of outcome
RateLimitApproved – Tracks requests that passed rate limiting checks
RateLimitDenied – Tracks requests blocked by rate limiting
InvocationFailed – Counts requests that failed during model invocation
InputTokens – Measures input token consumption for successful requests
OutputTokens – Measures output token consumption for successful requests
Each metric includes dimensions for Model, ModelId, CostCenter, Application, and Environment for data analysis.
We use CloudWatch metrics query capabilities with math expressions to analyze the data collected by the workflow. The data can be displayed in a variety of visual formats to get a granular view of requests by the dimensions provided, such as model or cost center. The following screenshot shows an example dashboard that displays invocation metrics where one model has reached its limit.

Additional Amazon Bedrock analytics
In addition to the custom metrics dashboard, CloudWatch provides automatic dashboards for monitoring Amazon Bedrock performance and usage. The Bedrock dashboard offers visibility into key performance metrics and operational insights, as shown in the following screenshot.

Cost tagging and reporting
Amazon Bedrock has introduced application inference profiles, a new capability that organizations can use to apply custom cost allocation tags to track and manage their on-demand foundation model (FM) usage. This feature addresses a previous limitation where tagging wasn’t possible for on-demand FMs, making it difficult to track costs across different business units and applications. You can now create custom inference profiles for base FMs and apply cost allocation tags like department, team, and application identifiers. These tags integrate with AWS cost management tools including AWS Cost Explorer, AWS Budgets, and AWS Cost Anomaly Detection, enabling detailed cost analysis and budget control.
Application inference profiles
To start, you must create application inference profiles for each type of usage you want to track. In this case, the solution defines custom tags for costCenter, environment, and applicationId. An inference profile will also be based on an existing Amazon Bedrock model profile, so you must combine the desired tags and model into the profile. At the time of writing, you must use the AWS Command Line Interface (AWS CLI) or AWS API to create one. See the following example code:

aws bedrock create-inference-profile
  –inference-profile-name “aws-docs-sales-prod”
  –model-source ‘{“copyFrom”:  “arn:aws:bedrock:us-east-1::foundation-model/anthropic.claude-3-haiku-20240307-v1:0”}’
  –tags ‘[
    {“key”: “applicationId”, “value”: “aws-documentation-helper”},
    {“key”: “costCenter”, “value”: “sales”},
    {“key”: “environment”, “value”: “production”}
  ]’

This command creates a profile for the sales cost center and production environment using Anthropic’s Claude Haiku 3.5 model. The output from this command is an Amazon Resource Name (ARN) that you will use as the model ID. In this solution, the ValidateAndSetContext Lambda function was modified to allow for specifying the model by cost center (for example, sales). To see which profiles you created, use the following command:
aws bedrock list-inference-profiles –type-equals APPLICATION
After the profiles have been created and the validation has been updated to map cost centers to the profile ARNs, the workflow will start running inference requests with the aligned profile. For example, when the user submits a request, they will specify the model as sales, services, or support to align with the three cost centers defined. The following code is a similar map to the previous example:

MODEL_ID_MAPPING = {
    “sales”: “arn:aws:bedrock:<region>:<account>:application-inference-profile/<unique id1>”,
    “services”: “arn:aws:bedrock:<region>:<account>:application-inference-profile/<unique id2>”,
    “support”: “arn:aws:bedrock:<region>:<account>:application-inference-profile/<unique id3>”
   }

To query CloudWatch metrics for the model usage correctly when using application inference profiles, you must specify the unique ID for the profile (the last part of the ARN). CloudWatch will store metrics like token usage based on the unique ID. To support both profile and direct model usage, the Lambda function was modified to add a new tag for modelMetric to be the appropriate term to use to query for token usage. See the following code:

  “tags”: {
    “requestId”: “ded98994-eb76-48d9-9dbc-f269541b5e49”,
    “timestamp”: “2025-01-31T14:05:26.854682”,
    “applicationId”: “aws-documentation-helper”,
    “costCenter”: “support”,
    “environment”: “production”,    
    “modelMetric”: “<unique id> | <model id>”
  }

Cost Explorer
Cost Explorer is a powerful cost management tool that provides comprehensive visualization and analysis of your cloud spending across AWS services, including Amazon Bedrock. It offers intuitive dashboards to track historical costs, forecast future expenses, and gain insights into your cloud consumption. With Cost Explorer, you can break down expenses by service, tags, and custom dimensions, for detailed financial analysis. The tool updates on a daily basis.
When you use application inference profiles with Amazon Bedrock, your AI service usage is automatically tagged and flows directly into Billing and Cost Management. These tags enable detailed cost tracking across different dimensions like cost center, application, and environment. This means you can generate reports that break down Amazon Bedrock AI expenses by specific business units, projects, or organizational hierarchies, providing clear visibility into your generative AI spending.
Cost allocation tags
Cost allocation tags are key-value pairs that help you categorize and track AWS resource costs across your organization. In the context of Amazon Bedrock, these tags can include attributes like application name, cost center, environment, or project ID. To activate a cost allocation tag, you must first enable it on the Billing and Cost Management console. After they’re activated, these tags will appear in your AWS Cost and Usage Report (CUR), helping you break down Amazon Bedrock expenses with granular detail.
To activate a cost allocation tag, complete the following steps:

On the Billing and Cost Management console, in the navigation pane, choose Cost Allocation Tags.
Locate your tag (for this example, it’s named costCenter) and choose Activate.
Confirm the activation.

After activation, the costCenter tag will appear in your CUR and will be used in Cost Explorer. It might take 24 hours for the tag to become fully active in your billing reports.

Cost Explorer reporting
To create an Amazon Bedrock usage report in Cost Explorer based on your tag, complete the following steps:

On the Billing and Cost Management console, choose Cost Explorer in the navigation pane.
Set your desired date range (relative time range or custom period).
Select Daily or Monthly granularity.
On the Group by dropdown menu, choose Tag.
Choose costCenter as the tag key.
Review the displayed Amazon Bedrock costs broken down by each unique cost center value.
Optionally, filter the values by applying a filter in the Filters section:

Choose Tag filter.
Choose the costCenter tag.
Choose specific cost center values you want to analyze.

The resulting report will provide a detailed view of Amazon Bedrock AI service expenses, helping you compare spending across different organizational units or projects with precision.

Summary
The AWS Cost and Usage Reports (including budgets) act as trailing edge indicators because they show what you’ve already spent on Amazon Bedrock after the fact. By blending real-time alerts from Step Functions with comprehensive cost reports, you can get a 360-degree view of your Amazon Bedrock usage. This reporting can alert you before you overspend and help you understand your actual consumption. This approach gives you the power to manage AI resources proactively, keeping your innovation budget on track and your projects running smoothly.
Try out this cost management approach for your own use case, and share your feedback in the comments.

About the Author
Jason Salcido is a Startups Senior Solutions Architect with nearly 30 years of experience pioneering innovative solutions for organizations from startups to enterprises. His expertise spans cloud architecture, serverless computing, machine learning, generative AI, and distributed systems. Jason combines deep technical knowledge with a forward-thinking approach to design scalable solutions that drive value, while translating complex concepts into actionable strategies.

As organizations embrace generative AI powered by Amazon Bedrock, they face the challenge of managing costs associated with the token-based pricing model. Amazon Bedrock offers a pay-as-you-go pricing structure that can potentially lead to unexpected and excessive bills if usage is not carefully monitored. Traditional methods of cost monitoring, such as budget alerts and cost anomaly detection, can help spot unexpectedly high usage but are reactive in nature. To address costs proactively, it is vital to use both leading and trailing indicators.
Leading indicators are predictive signals that help you anticipate future trends or potential issues before they fully materialize. These indicators provide proactive insights that allow for timely intervention. In contrast, trailing indicators are retrospective measurements that confirm what has already occurred. By understanding and tracking both types of indicators, organizations can develop more strategic and responsive decision-making processes.
In this two-part series, we introduce a comprehensive solution for proactively managing Amazon Bedrock inference costs. Our approach features a cost sentry mechanism designed to establish and enforce token usage limits, providing organizations with a robust framework for controlling generative AI expenses. In this post, we focus on core architecture, cost sentry design, token usage tracking, and initial budget enforcement strategies. In Part 2, we explore advanced monitoring techniques, custom tagging, reporting, and long-term cost optimization best practices. The goal is to deliver a predictable, cost-effective approach to Amazon Bedrock deployments that aligns with organizational financial constraints.
Solution overview
Amazon Bedrock is billed on a token usage-based policy with charges based on the input and output tokens used. The rate charged depends on the model used and AWS Region where inference is performed. Developers must implement robust token management strategies in their applications to help prevent runaway costs, making sure generative AI applications include circuit breakers and consumption limits that align with budgetary constraints.
To address this, you can configure Amazon CloudWatch alarms or monitor costs with billing alerts and budgets, but these mechanisms look at incurred costs or usage after the fact. Another option is the Generative AI Gateway Solution in the AWS Solutions Library, which uses LiteLLM to enforce budgetary limits for Amazon Bedrock and other model providers.
This solution was developed to identify a proactive, centralized mechanism that could limit the generative AI usage to a specific budget that can be adjusted. This approach uses serverless workflows and native Amazon Bedrock integration that offers less operational complexity while providing large-scale performance and scaling.
When building applications with Amazon Bedrock, it is common practice to access the service through a developed API, either synchronously through a REST API or asynchronously through a queuing system. The following diagram compares these architectures.

For synchronous interactions, clients make direct REST API calls to Amazon Bedrock, passing in the necessary parameters. In an asynchronous architecture, clients submit inference requests to a queue or message broker, such as Amazon Simple Queue Service (Amazon SQS). A backend processing system, often implemented as a serverless function or a containerized application, continuously monitors the queue and processes incoming requests. This approach decouples the client from the inference processing, enabling scalability and resilience in handling bursts of requests.
This solution is a centralized mechanism that can be used to interact with Amazon Bedrock to serve as a proactive cost sentry. It is designed using a serverless architecture that uses AWS Step Functions to orchestrate a workflow that validates token usage against configured limits before allowing Amazon Bedrock inference requests to proceed. This solution makes sure that generative AI applications stay within predefined budgetary boundaries, providing cost predictability and control.
The following diagram illustrates the architecture we build in this post.

The core components of this solution include:

Rate limiter workflow – A Step Functions workflow that retrieves current token usage metrics from CloudWatch, compares them against predefined limits stored in Amazon DynamoDB, and determines whether to proceed with or deny the Amazon Bedrock inference request.
Amazon Bedrock model router – A separate Step Functions state machine that acts as a centralized gateway for invoking various Amazon Bedrock models. This component abstracts the complexity of handling different I/O parameters required by each model.
Token usage tracking – Uses CloudWatch metrics integration with Amazon Bedrock to retrieve current token usage data for input and output tokens across all or specific models.
Budget configuration – Allows setting token usage limits on a per-model basis by storing the desired budget values in DynamoDB. A default limit can also be set to apply to models without specific budgets defined.
Cost and usage visibility – Provides visibility for AI usage with CloudWatch dashboards and cost over time reporting in AWS Cost Explorer.

The solution follows a serverless architecture approach, using managed AWS services like Step Functions, AWS Lambda, DynamoDB, and CloudWatch to provide a scalable, extensible, and cost-effective implementation.
The goal is to provide a proactive method of setting generative AI usage limits that operate as a leading indicator to limit usage:

Proactive budgeting – Enforces token usage limits before allowing inference requests, helping prevent accidental overspending
Model-specific budgets – Supports setting individual budgets for different Amazon Bedrock models based on their pricing and usage patterns
Default budget fallback – If no specific budget is defined for a model, a default limit can be applied to enable cost control
Monitoring – Uses CloudWatch metrics integration to track token usage, enabling accurate budget enforcement
Serverless architecture – Uses Step Functions, Lambda, DynamoDB, and CloudWatch for a scalable and cost-effective solution
Extensibility – The modular design allows for seamless integration of additional Amazon Bedrock models or alternative inference methods

Step Functions workflows
In this section, we explore how the solution uses Step Functions to implement rate limiting and model routing workflows.
Rate limiting workflow
The rate limiting workflow is designed to take a minimal JSON document as input with the following format:

{
  “modelId”: “string”,       // e.g. “anthropic.claude-3-sonnet-20240229-v1:0”
  “prompt”: {
    “messages”: [
      {
        “role”: “string”,    // “system”, “user”, or “assistant”
        “content”: “string”
      }
    ]
  }
}

This workflow is the core component that enforces budgetary controls. The key steps are as follows:

A Lambda function retrieves the start and end dates for the current month, which is used to query token usage metrics for the appropriate time range.
The workflow queries CloudWatch to retrieve the current month’s token usage metrics for the specified Amazon Bedrock model.
The workflow retrieves the configured token usage limit for the specified Amazon Bedrock model from DynamoDB. If no specific limit is found, it falls back to retrieving the default limit.
The workflow compares the current token usage against the configured limit to determine if the budget has been exceeded or not.
If the token usage is within the budget, this step invokes the Amazon Bedrock model router state machine to perform the actual inference request.
Depending on the outcome of the budget check, the workflow returns either the formatted inference result or an error indicating that the budget has been exceeded.

The following diagram illustrates the Step Functions workflow.

Amazon Bedrock model router workflow
The Amazon Bedrock model router workflow is a separate Step Functions state machine responsible for invoking the appropriate Amazon Bedrock model based on the request parameters. It abstracts the complexity of handling different I/O formats required by various Amazon Bedrock models and combines the result into a standardized format.
The key steps in the workflow include:

Based on the provided model ID, the workflow determines the specific Amazon Bedrock model to be invoked.
The workflow calls the appropriate Amazon Bedrock model with the required input parameters.
The workflow normalizes the output from the Amazon Bedrock model to a consistent format for further processing or returning to the client.
The workflow returns the transformed inference result to the calling workflow (budget sentry workflow).

The following diagram illustrates the Step Functions workflow.

You can implement additional steps to handle error conditions and format the output appropriately. In this example, the Anthropic flow includes error processing.
Token usage tracking with CloudWatch metrics
The Amazon Bedrock cost sentry uses the CloudWatch integration with Amazon Bedrock to retrieve current token usage metrics. These metrics are used to enforce budgetary limits proactively. For example, see the following query:

{
    “sparkline”: false,
    “metrics”: [
        [ { “expression”: “SEARCH(‘{AWS/Bedrock} MetricName=”InputTokenCount”‘, ‘Sum’, 60)”, “region”: “us-east-1” } ],
        [ { “expression”: “SEARCH(‘{AWS/Bedrock} MetricName=”OutputTokenCount”‘, ‘Sum’, 60)”, “region”: “us-east-1” } ]
    ],
    “legend”: {
        “position”: “right”
    },
    “title”: “InputTokenCount, OutputTokenCount”,
    “region”: “us-east-1”,
    “liveData”: true,
    “view”: “gauge”,
    “stacked”: false,
    “period”: 2592000,
    “table”: {
        “summaryColumns”: [
            “SUM”
        ]
    },
    “yAxis”: {
        “left”: {
            “min”: 0,
            “max”: 1000000
        }
    },
    “setPeriodToTimeRange”: true,
    “trend”: false,
    “startTime”: “2024-05-01T00:00:00Z”,
    “endTime”: “2024-05-30T23:59:59Z”
}

This CloudWatch metric query retrieves the total input and output token counts for a specified time range, allowing the rate limiter workflow to accurately enforce budgets based on real-time usage data.
Budget configuration with DynamoDB
The Amazon Bedrock cost sentry stores token usage limits in a DynamoDB table, providing seamless configuration and updates to individual model budgets or the default limit. For example, see the following code:

{
    “modelId”: “anthropic.claude-3-sonnet-20240229-v1:0”,
    “limit”: {
        “input”: 1000000,
        “output”: 3000000
    }
}

In this example, the token usage limit for the specified Amazon Bedrock model (anthropic.claude-3-sonnet-20240229-v1:0) is set to 1,000,000 input tokens and 3,000,000 output tokens.
Administrators can quickly update these limits by modifying the corresponding DynamoDB records, providing flexibility in adjusting budgets as needed.
Performance analysis of the rate limiter workflow
To assess the performance impact of introducing the workflow, we used an array of inference requests. Test cases included various prompts designed to generate responses ranging from concise answers to detailed explanations over 500 words, effectively testing the workflow’s performance across different output token sizes. The workflow demonstrated exceptional performance characteristics across 501 successful executions, handling a diverse set of inference requests from brief responses to extensive content generation.
The workflow maintains consistent execution patterns while processing requests ranging from 6.76 seconds to 32.24 seconds in total duration, with the variation primarily reflecting the different output token requirements of each request:

Quick responses (under 10 seconds) – Typically handling concise answers and simple queries
Medium-length content (11–22 seconds) – Common for detailed explanations and multi-paragraph responses
Extended generation (up to 32 seconds) – Handling comprehensive responses requiring more than 500 words

The following diagram illustrates our time distribution findings.

The time distribution analysis reveals highly optimized resource utilization:

Amazon Bedrock model router – 5.80–31.99 seconds (98.26% of runtime)
Other workflow steps – 0.11–4.74 seconds (1.65% of runtime)
System overhead – 0.02 seconds average (0.09% of runtime)

This performance profile aligns with best practices for workflow orchestration, where minimizing overhead and maintaining consistent execution patterns are crucial for reliability. The workflow’s efficiency is evidenced by its remarkably low system overhead of just 0.09%, demonstrating effective use of the built-in controls and state management capabilities of Step Functions regardless of the response size being generated.
The execution consistency is particularly noteworthy, with a predictable event pattern of 47–49 events per execution, regardless of the inference request complexity or output size. This predictability is essential for workload management and resource planning, especially when handling varied request complexities and token outputs.
These metrics indicate a well-architected workflow that effectively uses Step Functions Express workflow capabilities for high-volume event processing while maintaining minimal overhead and consistent performance characteristics across both simple queries and complex, token-intensive inference requests.
Cost analysis
To analyze the cost implications, estimates were generated using the AWS Pricing Calculator for both Standard and Express Step Functions workflows, assuming 100,000 requests per month. The following table summarizes these estimates.

Detailed Estimate

Region
Description
Service
Upfront
Monthly
First 12 Months Total
Currency
Configuration Summary

US East (Ohio)
Step Functions Standard
Step Functions – Standard Workflows
0
$37.40
$448.80
USD
Workflow requests (100,000 per month)State transitions per workflow (15)

US East (Ohio)
Step Functions Express
Step Functions – Express Workflows
0
$3.75
$45
USD
Duration of each workflow (35,000)Memory consumed by each workflow (64 MB)Workflow requests (100,000 per month)

The cost analysis revealed that the Step Functions Express workflow offers a more cost-effective solution compared to the Standard workflow, with potential cost savings of up to 90% for the same workload. There is a potential for cost reduction for Standard if the number of steps can be optimized. For example, a few formatting pass steps could potentially be removed, but these steps help format the downstream input to later steps.
Consult the AWS Pricing Calculator for more details on pricing and to run your own scenario.
Conclusion
In this solution, we used Step Functions to build a system that serves as a leading indicator because it tracks rate limiting and token usage, warning us immediately when we’re approaching our usage limits. In Part 2, we discuss combining this with trailing indicators to stay aware of usage and costs.

About the author
Jason Salcido is a Startups Senior Solutions Architect with nearly 30 years of experience pioneering innovative solutions for organizations from startups to enterprises. His expertise spans cloud architecture, serverless computing, machine learning, generative AI, and distributed systems. Jason combines deep technical knowledge with a forward-thinking approach to design scalable solutions that drive value, while translating complex concepts into actionable strategies.