Category: Uncategorized

In this tutorial, we explore the advanced capabilities of PyTest, one of the most powerful testing frameworks in Python. We build a complete mini-project from scratch that demonstrates fixtures, markers, plugins, parameterization, and custom configuration. We focus on showing how PyTest can evolve from a simple test runner into a robust, extensible system for real-world applications. By the end, we understand not just how to write tests, but how to control and customize PyTest’s behavior to fit any project’s needs. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserimport sys, subprocess, os, textwrap, pathlib, json

subprocess.run([sys.executable, “-m”, “pip”, “install”, “-q”, “pytest>=8.0”], check=True)

root = pathlib.Path(“pytest_advanced_tutorial”).absolute()
if root.exists():
import shutil; shutil.rmtree(root)
(root / “calc”).mkdir(parents=True)
(root / “app”).mkdir()
(root / “tests”).mkdir()

We begin by setting up our environment, importing essential Python libraries for file handling and subprocess execution. We install the latest version of PyTest to ensure compatibility and then create a clean project structure with folders for our main code, application modules, and tests. This gives us a solid foundation to organize everything neatly before writing any test logic. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browser(root / “pytest.ini”).write_text(textwrap.dedent(“””
[pytest]
addopts = -q -ra –maxfail=1 -m “not slow”
testpaths = tests
markers =
slow: slow tests (use –runslow to run)
io: tests hitting the file system
api: tests patching external calls
“””).strip()+”n”)

(root / “conftest.py”).write_text(textwrap.dedent(r”’
import os, time, pytest, json
def pytest_addoption(parser):
parser.addoption(“–runslow”, action=”store_true”, help=”run slow tests”)
def pytest_configure(config):
config.addinivalue_line(“markers”, “slow: slow tests”)
config._summary = {“passed”:0,”failed”:0,”skipped”:0,”slow_ran”:0}
def pytest_collection_modifyitems(config, items):
if config.getoption(“–runslow”):
return
skip = pytest.mark.skip(reason=”need –runslow to run”)
for item in items:
if “slow” in item.keywords: item.add_marker(skip)
def pytest_runtest_logreport(report):
cfg = report.config._summary
if report.when==”call”:
if report.passed: cfg[“passed”]+=1
elif report.failed: cfg[“failed”]+=1
elif report.skipped: cfg[“skipped”]+=1
if “slow” in report.keywords and report.passed: cfg[“slow_ran”]+=1
def pytest_terminal_summary(terminalreporter, exitstatus, config):
s=config._summary
terminalreporter.write_sep(“=”, “SESSION SUMMARY (custom plugin)”)
terminalreporter.write_line(f”Passed: {s[‘passed’]} | Failed: {s[‘failed’]} | Skipped: {s[‘skipped’]}”)
terminalreporter.write_line(f”Slow tests run: {s[‘slow_ran’]}”)
terminalreporter.write_line(“PyTest finished successfully ” if s[“failed”]==0 else “Some tests failed “)

@pytest.fixture(scope=”session”)
def settings(): return {“env”:”prod”,”max_retries”:2}
@pytest.fixture(scope=”function”)
def event_log(): logs=[]; yield logs; print(“\nEVENT LOG:”, logs)
@pytest.fixture
def temp_json_file(tmp_path):
p=tmp_path/”data.json”; p.write_text(‘{“msg”:”hi”}’); return p
@pytest.fixture
def fake_clock(monkeypatch):
t={“now”:1000.0}; monkeypatch.setattr(time,”time”,lambda: t[“now”]); return t
”’))

We now create our PyTest configuration and plugin files. In pytest.ini, we define markers, default options, and test paths to control how tests are discovered and filtered. In conftest.py, we implement a custom plugin that tracks passed, failed, and skipped tests, adds a –runslow option, and provides fixtures for reusable test resources. This helps us extend PyTest’s core behavior while keeping our setup clean and modular. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browser(root/”calc”/”__init__.py”).write_text(textwrap.dedent(”’
from .vector import Vector
def add(a,b): return a+b
def div(a,b):
if b==0: raise ZeroDivisionError(“division by zero”)
return a/b
def moving_avg(xs,k):
if k<=0 or k>len(xs): raise ValueError(“bad window”)
out=[]; s=sum(xs[:k]); out.append(s/k)
for i in range(k,len(xs)):
s+=xs[i]-xs[i-k]; out.append(s/k)
return out
”’))

(root/”calc”/”vector.py”).write_text(textwrap.dedent(”’
class Vector:
__slots__=(“x”,”y”,”z”)
def __init__(self,x=0,y=0,z=0): self.x,self.y,self.z=float(x),float(y),float(z)
def __add__(self,o): return Vector(self.x+o.x,self.y+o.y,self.z+o.z)
def __sub__(self,o): return Vector(self.x-o.x,self.y-o.y,self.z-o.z)
def __mul__(self,s): return Vector(self.x*s,self.y*s,self.z*s)
__rmul__=__mul__
def norm(self): return (self.x**2+self.y**2+self.z**2)**0.5
def __eq__(self,o): return abs(self.x-o.x)<1e-9 and abs(self.y-o.y)<1e-9 and abs(self.z-o.z)<1e-9
def __repr__(self): return f”Vector({self.x:.2f},{self.y:.2f},{self.z:.2f})”
”’))

We now build the core calculation module for our project. In the calc package, we define simple mathematical utilities, including addition, division with error handling, and a moving-average function, to demonstrate logic testing. Alongside this, we create a Vector class that supports arithmetic operations, equality checks, and norm computation, a perfect example for testing custom objects and comparisons using PyTest. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browser(root/”app”/”io_utils.py”).write_text(textwrap.dedent(”’
import json, pathlib, time
def save_json(path,obj):
path=pathlib.Path(path); path.write_text(json.dumps(obj)); return path
def load_json(path): return json.loads(pathlib.Path(path).read_text())
def timed_operation(fn,*a,**kw):
t0=time.time(); out=fn(*a,**kw); t1=time.time(); return out,t1-t0
”’))
(root/”app”/”api.py”).write_text(textwrap.dedent(”’
import os, time, random
def fetch_username(uid):
if os.environ.get(“API_MODE”)==”offline”: return f”cached_{uid}”
time.sleep(0.001); return f”user_{uid}_{random.randint(100,999)}”
”’))

(root/”tests”/”test_calc.py”).write_text(textwrap.dedent(”’
import pytest, math
from calc import add,div,moving_avg
from calc.vector import Vector
@pytest.mark.parametrize(“a,b,exp”,[(1,2,3),(0,0,0),(-1,1,0)])
def test_add(a,b,exp): assert add(a,b)==exp
@pytest.mark.parametrize(“a,b,exp”,[(6,3,2),(8,2,4)])
def test_div(a,b,exp): assert div(a,b)==exp
@pytest.mark.xfail(raises=ZeroDivisionError)
def test_div_zero(): div(1,0)
def test_avg(): assert moving_avg([1,2,3,4,5],3)==[2,3,4]
def test_vector_ops(): v=Vector(1,2,3)+Vector(4,5,6); assert v==Vector(5,7,9)
”’))

(root/”tests”/”test_io_api.py”).write_text(textwrap.dedent(”’
import pytest, os
from app.io_utils import save_json,load_json,timed_operation
from app.api import fetch_username
@pytest.mark.io
def test_io(temp_json_file,tmp_path):
d={“x”:5}; p=tmp_path/”a.json”; save_json(p,d); assert load_json(p)==d
assert load_json(temp_json_file)=={“msg”:”hi”}
def test_timed(capsys):
val,dt=timed_operation(lambda x:x*3,7); print(“dt=”,dt); out=capsys.readouterr().out
assert “dt=” in out and val==21
@pytest.mark.api
def test_api(monkeypatch):
monkeypatch.setenv(“API_MODE”,”offline”)
assert fetch_username(9)==”cached_9″
”’))

(root/”tests”/”test_slow.py”).write_text(textwrap.dedent(”’
import time, pytest
@pytest.mark.slow
def test_slow(event_log,fake_clock):
event_log.append(f”start@{fake_clock[‘now’]}”)
fake_clock[“now”]+=3.0
event_log.append(f”end@{fake_clock[‘now’]}”)
assert len(event_log)==2
”’))

We add lightweight app utilities for JSON I/O and a mocked API to exercise real-world behaviors without external services. We write focused tests that use parametrization, xfail, markers, tmp_path, capsys, and monkeypatch to validate logic and side effects. We include a slow test wired to our event_log and fake_clock fixtures to demonstrate controlled timing and session-wide state. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserprint(” Project created at:”, root)
print(“n RUN #1 (default, skips @slow)n”)
r1=subprocess.run([sys.executable,”-m”,”pytest”,str(root)],text=True)
print(“n RUN #2 (–runslow)n”)
r2=subprocess.run([sys.executable,”-m”,”pytest”,str(root),”–runslow”],text=True)

summary_file=root/”summary.json”
summary={
“total_tests”:sum(“test_” in str(p) for p in root.rglob(“test_*.py”)),
“runs”: [“default”,”–runslow”],
“results”: [“success” if r1.returncode==0 else “fail”,
“success” if r2.returncode==0 else “fail”],
“contains_slow_tests”: True,
“example_event_log”:[“start@1000.0″,”end@1003.0”]
}
summary_file.write_text(json.dumps(summary,indent=2))
print(“n FINAL SUMMARY”)
print(json.dumps(summary,indent=2))
print(“n Tutorial completed — all tests & summary generated successfully.”)

We now run our test suite twice: first with the default configuration that skips slow tests, and then again with the –runslow flag to include them. After both runs, we generate a JSON summary containing test outcomes, the total number of test files, and a sample event log. This final summary gives us a clear snapshot of our project’s testing health, confirming that all components work flawlessly from start to finish.

In conclusion, we see how PyTest helps us test smarter, not harder. We design a plugin that tracks results, uses fixtures for state management, and controls slow tests with custom options, all while keeping the workflow clean and modular. We conclude with a detailed JSON summary that demonstrates how easily PyTest can integrate with modern CI and analytics pipelines. With this foundation, we are now confident to extend PyTest further, combining coverage, benchmarking, or even parallel execution for large-scale, professional-grade testing.

Check out the FULL CODES here. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post A Coding Implementation of Advanced PyTest to Build Customized and Automated Testing with Plugins, Fixtures, and JSON Reporting appeared first on MarkTechPost.

Andrej Karpathy has open-sourced nanochat, a compact, dependency-light codebase that implements a full ChatGPT-style stack—from tokenizer training to web UI inference—aimed at reproducible, hackable LLM training on a single multi-GPU node.

The repo provides a single-script “speedrun” that executes the full loop: tokenization, base pretraining, mid-training on chat/multiple-choice/tool-use data, Supervised Finetuning (SFT), optional RL on GSM8K, evaluation, and serving (CLI + ChatGPT-like web UI). The recommended setup is an 8×H100 node; at ~$24/hour, the 4-hour speedrun lands near $100. A post-run report.md summarizes metrics (CORE, ARC-E/C, MMLU, GSM8K, HumanEval, ChatCORE).

Tokenizer and data path

Tokenizer: custom Rust BPE (built via Maturin), with a 65,536-token vocab; training uses FineWeb-EDU shards (re-packaged/shuffled for simple access). The walkthrough reports ~4.8 characters/token compression and compares against GPT-2/4 tokenizers.

Eval bundle: a curated set for CORE (22 autocompletion datasets like HellaSwag, ARC, BoolQ, etc.), downloaded into ~/.cache/nanochat/eval_bundle.

Model, scaling, and “speedrun” target

The speedrun config trains a depth-20 Transformer (≈560M params with 1280 hidden channels, 10 attention heads of dim 128) for ~11.2B tokens consistent with Chinchilla-style scaling (params × ~20 tokens). The author estimates this run as a ~4e19 FLOPs capability model. Training uses Muon for matmul parameters and AdamW for embeddings/unembeddings; loss is reported in bits-per-byte (bpb) to be tokenizer-invariant.

Mid-training, SFT, and tool use

After pretraining, mid-training adapts the base model to conversations (SmolTalk) and explicitly teaches multiple-choice behavior (100K MMLU auxiliary-train questions) and tool use by inserting <|python_start|>…<|python_end|> blocks; a small GSM8K slice is included to seed calculator-style usage. The default mixture: SmolTalk (460K), MMLU aux-train (100K), GSM8K main (8K), totaling 568K rows.

SFT then fine-tunes on higher-quality conversations while matching test-time formatting (padded, non-concatenated rows) to reduce train/inference mismatch. The repo’s example post-SFT metrics (speedrun tier) report ARC-Easy 0.3876, ARC-Challenge 0.2807, MMLU 0.3151, GSM8K 0.0455, HumanEval 0.0854, ChatCORE 0.0884.

Tool use is wired end-to-end: the custom Engine implements KV cache, prefill/decode inference, and a simple Python interpreter sandbox for tool-augmented runs—used in both training and evaluation flows.

Optional RL on GSM8K via a simplified GRPO loop

The final (optional) stage applies reinforcement learning on GSM8K with a simplified GRPO routine. The walkthrough clarifies what’s omitted relative to canonical PPO-style RLHF: no trust region via a reference model, no KL penalties, on-policy updates (discard PPO ratios/clip), token-level GAPO-style normalization, and mean-shift advantage. Practically, it behaves close to REINFORCE while keeping the group-relative advantage calculation. Scripts scripts.chat_rl and scripts.chat_eval -i rl -a GSM8K demonstrate the loop.

Cost/quality scaling and bigger models

The README sketches two larger targets beyond the ~$100 speedrun:

~$300 tier: d=26 (~12 hours), slightly surpasses GPT-2 CORE; requires more pretraining shards and batch-size adjustments.

~$1,000 tier: ~41.6 hours, with materially improved coherence and basic reasoning/coding ability.

The repo also note prior experimental runs where a d=30 model trained for ~24 hours reached 40s on MMLU, 70s on ARC-Easy, 20s on GSM8K.

Evaluation snapshot (speedrun tier)

An example report.md table for the ~$100/≈4-hour run shows: CORE 0.2219 (base); after mid-training/SFT, ARC-E 0.3561→0.3876, ARC-C ~0.2875→0.2807, MMLU 0.3111→0.3151, GSM8K 0.0250→0.0455, HumanEval 0.0671→0.0854, ChatCORE 0.0730→0.0884; wall-clock 3h51m.

https://github.com/karpathy/nanochat/discussions/1

Key Takeaways

nanochat is a minimal, end-to-end ChatGPT-style stack (~8K LOC) that runs via a single speedrun.sh on one 8×H100 node (~4h ≈ $100).

The pipeline covers tokenizer (Rust BPE), base pretraining, mid-training, SFT, optional RL on GSM8K (simplified GRPO), evaluation, and serving (CLI + Web UI).

Speedrun metrics (example report.md): CORE 0.2219 base; after SFT—ARC-Easy 0.3876, ARC-Challenge 0.2807, MMLU 0.3151, GSM8K 0.0455, HumanEval 0.0854.

Scaling tiers are outlined: ~$300 (d=26, ~12h) “slightly outperforms GPT-2 CORE”; ~$1,000 (~41.6h) for materially better coherence/reasoning.

Editorial Comments

Karpathy’s nanochat lands in a useful middle ground: a single, clean, dependency-light repository that stitches tokenizer training (Rust BPE), pretraining on FineWeb-EDU, mid-training (SmolTalk/MMLU aux/GSM8K with tool use tags), SFT, optional simplified GRPO on GSM8K, and a thin Engine (KV cache, prefill/decode, Python interpreter) into a reproducible speedrun on an 8×H100 node, producing a traceable report.md with CORE/ARC/MMLU/GSM8K/HumanEval and a minimal Web UI.

Check out the Technical details and Codes. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.

Excited to release new repo: nanochat!(it’s among the most unhinged I’ve written).Unlike my earlier similar repo nanoGPT which only covered pretraining, nanochat is a minimal, from scratch, full-stack training/inference pipeline of a simple ChatGPT clone in a single,… pic.twitter.com/LLhbLCoZFt— Andrej Karpathy (@karpathy) October 13, 2025

The post Andrej Karpathy Releases ‘nanochat’: A Minimal, End-to-End ChatGPT-Style Pipeline You Can Train in ~4 Hours for ~$100 appeared first on MarkTechPost.

The proliferation of Internet of Things (IoT) devices has transformed how we interact with our environments, from homes to industrial settings. However, as the number of connected devices grows, so does the complexity of managing them. Traditional device management interfaces often require navigating through multiple applications, each with its own UI and learning curve. This fragmentation creates friction for users trying to monitor and control their IoT environment.
In this post, we explore how to build a conversational device management system using Amazon Bedrock AgentCore. With this solution, users can manage their IoT devices through natural language, using a UI for tasks like checking device status, configuring WiFi networks, and monitoring user activity. To learn more about how Amazon Bedrock AgentCore enables deploying and operating highly effective agents securely at scale using a variety of frameworks and models, refer to Enabling customers to deliver production-ready AI agents at scale.
The challenge of device management
Managing a modern IoT environment involves navigating numerous challenges that can hinder user experience and technology adoption. Interface fragmentation forces users to juggle multiple applications and management tools for different devices, and technical complexity can make even basic configuration tasks intimidating for non-specialists. Adding to these difficulties are visibility limitations that prevent comprehensive monitoring of device status, and inadequate user management capabilities that make it difficult to track device usage patterns.
Together, these pain points create significant friction for users trying to implement and maintain IoT solutions effectively.
Solution overview
The conversational AI solution using agents offers a comprehensive approach to IoT complexity through its unified conversational interface that consolidates device management tasks into a single access point. Users can perform sophisticated operations through natural language interaction instead of navigating technical menus, while gaining comprehensive visibility across connected devices and transforming complex configuration tasks into straightforward conversations. The system delivers essential capabilities, including device management for inventory control and status monitoring, WiFi network management for simplified network configuration, user management for access control, and activity tracking for temporal analysis of user interactions. This seamless management experience minimizes monitoring vulnerabilities and provides valuable insights into usage patterns and potential security concerns, effectively removing the typical barriers to successful IoT implementation while maintaining appropriate system authorization throughout the network.
Architecture overview

The device management system follows a modular architecture that uses several AWS services. The architecture consists of the following components:

User and application interface – Users interact with the system through a web application that serves as the frontend interface.
Foundation models – This system uses various foundation models (FMs) in Amazon Bedrock to power natural language understanding and generation capabilities.
Amazon Bedrock AgentCore Gateway – This feature acts as the secure entry point for authenticated requests, validating bearer tokens before routing requests to the appropriate target.
Amazon Bedrock AgentCore Identity – This feature manages agent identity and permissions, controlling what actions the agent can perform on behalf of users.
Amazon Bedrock AgentCore Memory – This feature supports both short-term and long-term memory, maintaining immediate conversation context within a session and storing persistent insights and preferences across sessions. This enables agents to provide consistent, context-aware responses without developers needing to manage complex memory infrastructure.
Amazon Bedrock AgentCore Observability – This feature monitors agent performance, tracks metrics, and provides insights into system usage and behavior for debugging and optimization.
Amazon Bedrock AgentCore Runtime – This secure, serverless environment supports AI agents built with open source frameworks. It maintains complete session isolation by dedicating isolated containers per user session, enabling scalable and secure management of long-running, stateful interactions.
Amazon Cognito – Amazon Cognito handles user authentication through bearer token generation and validation, facilitating secure access to the system.
Amazon DynamoDB – Amazon DynamoDB stores system data across five tables.
AWS Lambda – The solution connects the gateway to AWS Lambda functions that execute specific device management operations. Lambda contains the business logic for device management, implementing seven core tools.

This architecture enables a seamless flow from user query to response: the user submits a natural language request through the application, which is authenticated through Amazon Cognito and processed by Amazon Bedrock AgentCore Runtime. The runtime determines the appropriate tool to invoke and sends the request through the gateway to the Lambda function, which queries or updates DynamoDB as needed. The result flows back through the same path, with the runtime generating a natural language response based on the data retrieved.
Refer to the GitHub repository for detailed deployment instructions.
Key functionalities of the device management agent
The device management system uses Lambda to implement seven essential tools for device management, including listing devices, retrieving settings, managing WiFi networks, and monitoring user activity, all invoked by the agent as needed. This functionality is supported by our flexible NoSQL database architecture in DynamoDB, which comprises five distinct tables—Devices, DeviceSettings, WifiNetworks, Users, and UserActivities—storing specialized data to maintain comprehensive system records. Together, these components create a robust foundation that enables efficient device management while maintaining detailed audit trails of system activities.
Key features showcase

Performance and security considerations
The solution balances robust concurrent processing capabilities with comprehensive protection measures. The device management system efficiently handles multiple simultaneous requests through automatically scaling Lambda functions, consistent DynamoDB performance regardless of data volume, and intelligent retry logic with exponential backoff when encountering rate limitations. To scale across hundreds of tools, the semantic search capability in Amazon Bedrock AgentCore Gateway enables efficient and relevant discovery of tools by meaning, facilitating quick and accurate responses even at large scale.
The system implements industry-leading security practices, including Amazon Cognito authentication, Amazon Bedrock AgentCore Identity, layered access control through gateway and Lambda level permission verification, comprehensive data encryption at rest and in transit, and Amazon Bedrock Guardrails to help prevent prompt injection attacks while maintaining interaction safety.
Conclusion
The device management system presented in this post uses Amazon Bedrock AgentCore to transform IoT management through conversational AI, creating an intuitive interface where complex device operations become simple dialogue. Its composable, reusable, and decoupled agentic architecture alleviates undifferentiated heavy lifting by providing built-in features for secure, scalable deployment and seamless integration. By combining large language models with an AWS infrastructure, the solution provides enterprise-grade capabilities without burdening developers with infrastructure management. Key benefits include simplified user experiences through natural language interaction, operational efficiency with unified interfaces, comprehensive device visibility, and future-proof architecture that evolves with AI advancements. The system’s model-agnostic approach supports continuous improvement as new FMs emerge, and robust security and observability features help organizations confidently deploy scalable, next-generation device management solutions tailored to their specific IoT environments.
To implement this solution, refer to the GitHub repository.

About the Author
Godwin Sahayaraj Vincent is an Enterprise Solutions Architect at AWS who is passionate about Machine Learning and providing guidance to customers to design, deploy and manage their AWS workloads and architectures. In his spare time, he loves to play cricket with his friends and tennis with his three kids.
Ramesh Kumar Venkatraman is a Senior Solutions Architect at AWS who is passionate about Generative AI, Containers and Databases. He works with AWS customers to design, deploy and manage their AWS workloads and architectures. In his spare time, he loves to play with his two kids and follows cricket.
Chhavi Kaushik is an AWS Solutions Architect specializing in cloud-native architectures and digital transformation. She is passionate about helping customers harness the power of Generative AI, designing and implementing enterprise-scale solutions that combine AWS’s cutting-edge AI/ML services. Outside of her professional life, Chhavi enjoys exploring the California outdoors, making the most of the Bay Area’s beautiful weather and lifestyle.

This post is cowritten by Salesforce’s AI Platform team members Srikanta Prasad, Utkarsh Arora, Raghav Tanaji, Nitin Surya, Gokulakrishnan Gopalakrishnan, and Akhilesh Deepak Gotmare.
Salesforce’s Artificial Intelligence (AI) platform team runs customized large language models (LLMs)—fine-tuned versions of Llama, Qwen, and Mistral—for agentic AI applications like Agentforce. Deploying these models creates operational overheads: teams spend months optimizing instance families, serving engines, and configurations. This process is time-consuming, hard to maintain with frequent releases, and expensive due to GPU capacity reservations for peak usage.
Salesforce solved this by adopting Amazon Bedrock Custom Model Import. With Amazon Bedrock Custom Model Import, teams can import and deploy customized models through a unified API, minimizing infrastructure management while integrating with Amazon Bedrock features like Amazon Bedrock Knowledge Bases, Amazon Bedrock Guardrails, and Amazon Bedrock Agents. This shift lets Salesforce focus on models and business logic instead of infrastructure.
This post shows how Salesforce integrated Amazon Bedrock Custom Model Import into their machine learning operations (MLOps) workflow, reused existing endpoints without application changes, and benchmarked scalability. We share key metrics on operational efficiency and cost optimization gains, and offer practical insights for simplifying your deployment strategy.
Integration approach
Salesforce’s transition from Amazon SageMaker Inference to Amazon Bedrock Custom Model Import required careful integration with their existing MLOps pipeline to avoid disrupting production workloads. The team’s primary goal was to maintain their current API endpoints and model serving interfaces, keeping zero downtime and no required changes to downstream applications. With this approach, they could use the serverless capabilities of Amazon Bedrock while preserving the investment in their existing infrastructure and tooling. The integration strategy focused on creating a seamless bridge between their current deployment workflows and Amazon Bedrock managed services, enabling gradual migration without additional operational risk.
As shown in the following deployment flow diagram, Salesforce enhanced their existing model delivery pipeline with a single additional step to use Amazon Bedrock Custom Model Import. After their continuous integration and continuous delivery (CI/CD) process saves model artifacts to their model store (an Amazon Simple Storage Service (Amazon S3) bucket), they now call the Amazon Bedrock Custom Model Import API to register the model with Amazon Bedrock. This control plane operation is lightweight because Amazon Bedrock pulls the model directly from Amazon S3, adding minimal overhead (5–7 mins, depending on model size) to their deployment timeline—their overall model release process remains at approximately 1 hour. The integration delivered an immediate performance benefit: SageMaker no longer needs to download weights at container startup because Amazon Bedrock preloads the model. The main configuration changes involved granting Amazon Bedrock permissions to allow cross-account access to their S3 model bucket and updating AWS Identity and Access Management (IAM) policies to allow inference clients to invoke Amazon Bedrock endpoints.

The following inference flow diagram illustrates how Salesforce maintained their existing application interfaces while using Amazon Bedrock serverless capabilities. Client requests flow through their established preprocessing layer for business logic like prompt formatting before reaching Amazon Bedrock, with postprocessing applied to the raw model output. To handle complex processing requirements, they deployed lightweight SageMaker CPU containers that act as intelligent proxies—running their custom model.py logic while forwarding the actual inference to Amazon Bedrock endpoints. This hybrid architecture preserves their existing tooling framework: their prediction service continues calling SageMaker endpoints without routing changes, and they retain mature SageMaker monitoring and logging for preprocessing and postprocessing logic. The trade-off involves an additional network hop adding 5–10 millisecond latency and the cost of always-on CPU instances, but this approach delivers backward-compatibility with existing integrations while keeping the GPU-intensive inference fully serverless through Amazon Bedrock.

Scalability benchmarking
To validate the performance capabilities of Amazon Bedrock Custom Model Import, Salesforce conducted comprehensive load testing across various concurrency scenarios. Their testing methodology focused on measuring how the transparent auto scaling behavior of Amazon Bedrock—where the service automatically spins up model copies on-demand and scales out under heavy load—would impact real-world performance. Each test involved sending standardized payloads containing model IDs and input data through their proxy containers to Amazon Bedrock endpoints, measuring latency and throughput under different load patterns. Results (see the following table) show that at low concurrency, Amazon Bedrock achieved 44% lower latency than the ml.g6e.xlarge baseline (bf16 precision). Under higher loads, Amazon Bedrock Custom Model Import maintained consistent throughput with acceptable latency (less than 10 milliseconds), demonstrating the serverless architecture’s ability to handle production workloads without manual scaling.

Concurrency (Count)
P95 Latency (in Seconds)
Throughput (Request per Minute)

1
7.2
11

4
7.96
41

16
9.35
133

32
10.44
232

The results show P95 latency and throughput performance of the ApexGuru model (fine-tuned QWEN-2.5 13B) at varying concurrency levels. Amazon Bedrock Custom Model Import auto scaled from one to three copies as concurrency reached 32. Each model copy used 1 model unit.
Results and metrics
Beyond scalability improvements, Salesforce evaluated Amazon Bedrock Custom Model Import across two critical business dimensions: operational efficiency and cost optimization. The operational efficiency gains were substantial—the team achieved a 30% reduction in time to iterate and deploy models to production. This improvement stemmed from alleviating complex decision-making around instance selection, parameter tuning, and choosing between serving engines like vLLM vs. TensorRT-LLM. The streamlined deployment process allowed developers to focus on model performance rather than infrastructure configuration.
Cost optimization delivered even more dramatic results, with Salesforce achieving up to 40% cost reduction through Amazon Bedrock. This savings was primarily driven by their diverse traffic patterns across generative AI applications—ranging from low to high production traffic—where they previously had to reserve GPU capacity for peak workloads. The pay-per-use model proved especially beneficial for development, performance testing, and staging environments that only required GPU resources during active development cycles, avoiding the need for round-the-clock reserved capacity that often sat idle.
Lessons learned
Salesforce’s journey with Amazon Bedrock Custom Model Import revealed several key insights that can guide other organizations considering a similar approach. First, although Amazon Bedrock Custom Model Import supports popular open source model architectures (Qwen, Mistral, Llama) and expands its portfolio frequently based on demand, teams working with cutting-edge architectures might need to wait for support. However, organizations fine-tuning with the latest model architectures should verify compatibility before committing to the deployment timeline.
For pre- and post-inference processing, Salesforce evaluated alternative approaches using Amazon API Gateway and AWS Lambda functions, which offer complete serverless scaling and pay-per-use pricing down to milliseconds of execution. However, they found this approach less backward-compatible with existing integrations and observed cold start impacts when using larger libraries in their processing logic.
Cold start latency emerged as a critical consideration, particularly for larger (over 7B parameter) models. Salesforce observed cold start delays of a couple of minutes with 26B parameter models, with latency varying based on model size. For latency-sensitive applications that can’t tolerate such delays, they recommend keeping endpoints warm by maintaining at least one model copy active through health check invocations every 14 minutes. This approach balances cost-efficiency with performance requirements for production workloads.
Conclusion
Salesforce’s adoption of Amazon Bedrock Custom Model Import shows how to simplify LLM deployment without sacrificing scalability or performance. They achieved 30% faster deployments and 40% cost savings while maintaining backward-compatibility through their hybrid architecture using SageMaker proxy containers alongside Amazon Bedrock serverless inference. For highly customized models or unsupported architectures, Salesforce continues using SageMaker AI as a managed ML solution.
Their success came from methodical execution: thorough load testing, and gradual migration starting with non-critical workloads. The results prove serverless AI deployment works for production, especially with variable traffic patterns. ApexGuru is now deployed in their production environment.
For teams managing LLMs at scale, this case study provides a clear blueprint. Check your model architecture compatibility, plan for cold starts with larger models, and preserve existing interfaces. Amazon Bedrock Custom Model Import offers a proven path to serverless AI that can reduce overhead, speed deployment, and cut costs while meeting performance requirements.
To learn more about pricing for Amazon Bedrock, refer to Optimizing cost for using foundational models with Amazon Bedrock and Amazon Bedrock pricing.
For help choosing between Amazon Bedrock and SageMaker AI, see Amazon Bedrock or Amazon SageMaker AI?
For more information about Amazon Bedrock Custom Model Import, see How to configure cross-account model deployment using Amazon Bedrock Custom Model Import.
For more details about ApexGuru, refer to Get AI-Powered Insights for Your Apex Code with ApexGuru.

About the authors
Srikanta Prasad is a Senior Manager in Product Management specializing in generative AI solutions at Salesforce. He leads Model Hosting and Inference initiatives, focusing on LLM inference serving, LLMOps, and scalable AI deployments.
Utkarsh Arora is an Associate Member of Technical Staff at Salesforce, combining strong academic grounding from IIIT Delhi with early career contributions in ML engineering and research. 
Raghav Tanaji is a Lead Member of Technical Staff at Salesforce, specializing in machine learning, pattern recognition, and statistical learning. He holds an M.Tech from IISc Bangalore.
Akhilesh Deepak Gotmare is a Senior Research Staff Member at Salesforce Research, based in Singapore. He is an AI Researcher focusing on deep learning, natural language processing, and code-related applications
Gokulakrishnan Gopalakrishnan is a Principal Software Engineer at Salesforce, where he leads engineering efforts on ApexGuru. With 15+ years of experience, including at Microsoft, he specializes in building scalable software systems
Nitin Surya is a Lead Member of Technical Staff at Salesforce with 8+ years in software/ML engineering. He holds a B.Tech in CS from VIT University and MS in CS (AI/ML focus) from University of Illinois Chicago.
Hrushikesh Gangur is a Principal Solutions Architect at AWS based in San Francisco, California. He specializes in generative and agentic AI, helping startups and ISVs build and deploy AI applications.

In this tutorial, we explore Ivy’s remarkable ability to unify machine learning development across frameworks. We begin by writing a fully framework-agnostic neural network that runs seamlessly on NumPy, PyTorch, TensorFlow, and JAX. We then dive into code transpilation, unified APIs, and advanced features like Ivy Containers and graph tracing, all designed to make deep learning code portable, efficient, and backend-independent. As we progress, we witness how Ivy simplifies model creation, optimization, and benchmarking without locking us into any single ecosystem. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browser!pip install -q ivy tensorflow torch jax jaxlib

import ivy
import numpy as np
import time

print(f”Ivy version: {ivy.__version__}”)

class IvyNeuralNetwork:
“””A simple neural network written purely in Ivy that works with any backend.”””

def __init__(self, input_dim=4, hidden_dim=8, output_dim=3):
self.w1 = ivy.random_uniform(shape=(input_dim, hidden_dim), low=-0.5, high=0.5)
self.b1 = ivy.zeros((hidden_dim,))
self.w2 = ivy.random_uniform(shape=(hidden_dim, output_dim), low=-0.5, high=0.5)
self.b2 = ivy.zeros((output_dim,))

def forward(self, x):
“””Forward pass using pure Ivy operations.”””
h = ivy.matmul(x, self.w1) + self.b1
h = ivy.relu(h)

out = ivy.matmul(h, self.w2) + self.b2
return ivy.softmax(out)

def train_step(self, x, y, lr=0.01):
“””Simple training step with manual gradients.”””
pred = self.forward(x)

loss = -ivy.mean(ivy.sum(y * ivy.log(pred + 1e-8), axis=-1))

pred_error = pred – y

h_activated = ivy.relu(ivy.matmul(x, self.w1) + self.b1)
h_t = ivy.permute_dims(h_activated, axes=(1, 0))
dw2 = ivy.matmul(h_t, pred_error) / x.shape[0]
db2 = ivy.mean(pred_error, axis=0)

self.w2 = self.w2 – lr * dw2
self.b2 = self.b2 – lr * db2

return loss

def demo_framework_agnostic_network():
“””Demonstrate the same network running on different backends.”””
print(“n” + “=”*70)
print(“PART 1: Framework-Agnostic Neural Network”)
print(“=”*70)

X = np.random.randn(100, 4).astype(np.float32)
y = np.eye(3)[np.random.randint(0, 3, 100)].astype(np.float32)

backends = [‘numpy’, ‘torch’, ‘tensorflow’, ‘jax’]
results = {}

for backend in backends:
try:
ivy.set_backend(backend)

if backend == ‘jax’:
import jax
jax.config.update(‘jax_enable_x64’, True)

print(f”n Running with {backend.upper()} backend…”)

X_ivy = ivy.array(X)
y_ivy = ivy.array(y)

net = IvyNeuralNetwork()

start_time = time.time()
for epoch in range(50):
loss = net.train_step(X_ivy, y_ivy, lr=0.1)

elapsed = time.time() – start_time

predictions = net.forward(X_ivy)
accuracy = ivy.mean(
ivy.astype(ivy.argmax(predictions, axis=-1) == ivy.argmax(y_ivy, axis=-1), ‘float32’)
)

results[backend] = {
‘loss’: float(ivy.to_numpy(loss)),
‘accuracy’: float(ivy.to_numpy(accuracy)),
‘time’: elapsed
}

print(f” Final Loss: {results[backend][‘loss’]:.4f}”)
print(f” Accuracy: {results[backend][‘accuracy’]:.2%}”)
print(f” Time: {results[backend][‘time’]:.3f}s”)

except Exception as e:
print(f” {backend} error: {str(e)[:80]}”)
results[backend] = None

ivy.unset_backend()
return results

We build and train a simple neural network entirely with Ivy to demonstrate true framework-agnostic design. We run the same model seamlessly across NumPy, PyTorch, TensorFlow, and JAX backends, observing consistent behavior and performance. Through this, we experience how Ivy abstracts away framework differences while maintaining efficiency and accuracy. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserdef demo_transpilation():
“””Demonstrate transpiling code from PyTorch to TensorFlow and JAX.”””
print(“n” + “=”*70)
print(“PART 2: Framework Transpilation”)
print(“=”*70)

try:
import torch
import tensorflow as tf

def pytorch_computation(x):
“””A simple PyTorch computation.”””
return torch.mean(torch.relu(x * 2.0 + 1.0))

x_torch = torch.randn(10, 5)

print(“n Original PyTorch function:”)
result_torch = pytorch_computation(x_torch)
print(f” PyTorch result: {result_torch.item():.6f}”)

print(“n Transpilation Demo:”)
print(” Note: ivy.transpile() is powerful but complex.”)
print(” It works best with traced/compiled functions.”)
print(” For simple demonstrations, we’ll show the unified API instead.”)

print(“n Equivalent computation across frameworks:”)
x_np = x_torch.numpy()

ivy.set_backend(‘numpy’)
x_ivy = ivy.array(x_np)
result_np = ivy.mean(ivy.relu(x_ivy * 2.0 + 1.0))
print(f” NumPy result: {float(ivy.to_numpy(result_np)):.6f}”)

ivy.set_backend(‘tensorflow’)
x_ivy = ivy.array(x_np)
result_tf = ivy.mean(ivy.relu(x_ivy * 2.0 + 1.0))
print(f” TensorFlow result: {float(ivy.to_numpy(result_tf)):.6f}”)

ivy.set_backend(‘jax’)
import jax
jax.config.update(‘jax_enable_x64’, True)
x_ivy = ivy.array(x_np)
result_jax = ivy.mean(ivy.relu(x_ivy * 2.0 + 1.0))
print(f” JAX result: {float(ivy.to_numpy(result_jax)):.6f}”)

print(f”n All results match within numerical precision!”)

ivy.unset_backend()

except Exception as e:
print(f” Demo error: {str(e)[:80]}”)

In this part, we explore how Ivy enables smooth transpilation and interoperability between frameworks. We take a simple PyTorch computation and reproduce it identically in TensorFlow, NumPy, and JAX using Ivy’s unified API. Through this, we see how Ivy bridges framework boundaries, enabling consistent results across different deep learning ecosystems. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserdef demo_unified_api():
“””Show how Ivy’s unified API works across different operations.”””
print(“n” + “=”*70)
print(“PART 3: Unified API Across Frameworks”)
print(“=”*70)

operations = [
(“Matrix Multiplication”, lambda x: ivy.matmul(x, ivy.permute_dims(x, axes=(1, 0)))),
(“Element-wise Operations”, lambda x: ivy.add(ivy.multiply(x, x), 2)),
(“Reductions”, lambda x: ivy.mean(ivy.sum(x, axis=0))),
(“Neural Net Ops”, lambda x: ivy.mean(ivy.relu(x))),
(“Statistical Ops”, lambda x: ivy.std(x)),
(“Broadcasting”, lambda x: ivy.multiply(x, ivy.array([1.0, 2.0, 3.0, 4.0]))),
]

X = np.random.randn(5, 4).astype(np.float32)

for op_name, op_func in operations:
print(f”n {op_name}:”)

for backend in [‘numpy’, ‘torch’, ‘tensorflow’, ‘jax’]:
try:
ivy.set_backend(backend)

if backend == ‘jax’:
import jax
jax.config.update(‘jax_enable_x64’, True)

x_ivy = ivy.array(X)
result = op_func(x_ivy)
result_np = ivy.to_numpy(result)

if result_np.shape == ():
print(f” {backend:12s}: scalar value = {float(result_np):.4f}”)
else:
print(f” {backend:12s}: shape={result_np.shape}, mean={np.mean(result_np):.4f}”)

except Exception as e:
print(f” {backend:12s}: {str(e)[:60]}”)

ivy.unset_backend()

In this section, we test Ivy’s unified API by performing various mathematical, neural, and statistical operations across multiple backends. We seamlessly execute the same code on NumPy, PyTorch, TensorFlow, and JAX, confirming consistent results and syntax. Through this, we realize how Ivy simplifies multi-framework coding into a single, coherent interface that just works everywhere. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserdef demo_advanced_features():
“””Demonstrate advanced Ivy features.”””
print(“n” + “=”*70)
print(“PART 4: Advanced Ivy Features”)
print(“=”*70)

print(“n Ivy Containers – Nested Data Structures:”)
try:
ivy.set_backend(‘torch’)

container = ivy.Container({
‘layer1’: {‘weights’: ivy.random_uniform(shape=(4, 8)), ‘bias’: ivy.zeros((8,))},
‘layer2’: {‘weights’: ivy.random_uniform(shape=(8, 3)), ‘bias’: ivy.zeros((3,))}
})

print(f” Container keys: {list(container.keys())}”)
print(f” Layer1 weight shape: {container[‘layer1’][‘weights’].shape}”)
print(f” Layer2 bias shape: {container[‘layer2’][‘bias’].shape}”)

def scale_fn(x, _):
return x * 2.0

scaled_container = container.cont_map(scale_fn)
print(f” Applied scaling to all tensors in container”)

except Exception as e:
print(f” Container demo: {str(e)[:80]}”)

print(“n Array API Standard Compliance:”)
backends_tested = []
for backend in [‘numpy’, ‘torch’, ‘tensorflow’, ‘jax’]:
try:
ivy.set_backend(backend)

if backend == ‘jax’:
import jax
jax.config.update(‘jax_enable_x64’, True)

x = ivy.array([1.0, 2.0, 3.0])
y = ivy.array([4.0, 5.0, 6.0])

result = ivy.sqrt(ivy.square(x) + ivy.square(y))
print(f” {backend:12s}: L2 norm operations work “)
backends_tested.append(backend)
except Exception as e:
print(f” {backend:12s}: {str(e)[:50]}”)

print(f”n Successfully tested {len(backends_tested)} backends”)

print(“n Complex Multi-step Operations:”)
try:
ivy.set_backend(‘torch’)

x = ivy.random_uniform(shape=(10, 5), low=0, high=1)

result = ivy.mean(
ivy.relu(
ivy.matmul(x, ivy.permute_dims(x, axes=(1, 0)))
),
axis=0
)

print(f” Chained operations (matmul → relu → mean)”)
print(f” Input shape: (10, 5), Output shape: {result.shape}”)
print(f” Complex operation graph executed successfully”)

except Exception as e:
print(f” {str(e)[:80]}”)

ivy.unset_backend()

We dive into Ivy’s power features beyond the basics. We organize parameters with ivy.Container, validate Array API–style ops across NumPy, PyTorch, TensorFlow, and JAX, and chain complex steps (matmul → ReLU → mean) to see graph-like execution flow. We come away confident that Ivy scales from neat data structures to robust multi-backend computation. Check out the FULL CODES here.

Copy CodeCopiedUse a different Browserdef benchmark_operation(op_func, x, iterations=50):
“””Benchmark an operation.”””
start = time.time()
for _ in range(iterations):
result = op_func(x)
return time.time() – start

def demo_performance():
“””Compare performance across backends.”””
print(“n” + “=”*70)
print(“PART 5: Performance Benchmarking”)
print(“=”*70)

X = np.random.randn(100, 100).astype(np.float32)

def complex_operation(x):
“””A more complex computation.”””
z = ivy.matmul(x, ivy.permute_dims(x, axes=(1, 0)))
z = ivy.relu(z)
z = ivy.mean(z, axis=0)
return ivy.sum(z)

print(“n Benchmarking matrix operations (50 iterations):”)
print(” Operation: matmul → relu → mean → sum”)

for backend in [‘numpy’, ‘torch’, ‘tensorflow’, ‘jax’]:
try:
ivy.set_backend(backend)

if backend == ‘jax’:
import jax
jax.config.update(‘jax_enable_x64’, True)

x_ivy = ivy.array(X)

_ = complex_operation(x_ivy)

elapsed = benchmark_operation(complex_operation, x_ivy, iterations=50)

print(f” {backend:12s}: {elapsed:.4f}s ({elapsed/50*1000:.2f}ms per op)”)

except Exception as e:
print(f” {backend:12s}: {str(e)[:60]}”)

ivy.unset_backend()

if __name__ == “__main__”:
print(“””
╔════════════════════════════════════════════════════════════════════╗
║ Advanced Ivy Tutorial – Framework-Agnostic ML ║
║ Write Once, Run Everywhere! ║
╚════════════════════════════════════════════════════════════════════╝
“””)

results = demo_framework_agnostic_network()
demo_transpilation()
demo_unified_api()
demo_advanced_features()
demo_performance()

print(“n” + “=”*70)
print(” Tutorial Complete!”)
print(“=”*70)
print(“n Key Takeaways:”)
print(” 1. Ivy enables writing ML code once that runs on any framework”)
print(” 2. Same operations work identically across NumPy, PyTorch, TF, JAX”)
print(” 3. Unified API provides consistent operations across backends”)
print(” 4. Switch backends dynamically for optimal performance”)
print(” 5. Containers help manage complex nested model structures”)
print(“n Next Steps:”)
print(” – Build your own framework-agnostic models”)
print(” – Use ivy.Container for managing model parameters”)
print(” – Explore ivy.trace_graph() for computation graph optimization”)
print(” – Try different backends to find optimal performance”)
print(” – Check docs at: https://docs.ivy.dev/”)
print(“=”*70)

We benchmark the same complex operation across NumPy, PyTorch, TensorFlow, and JAX to compare real-world throughput. We warm up each backend, run 50 iterations, and log total time and per-op latency so we can choose the fastest stack for our workload.

In conclusion, we experience firsthand how Ivy empowers us to “write once and run everywhere.” We observe identical model behavior, seamless backend switching, and consistent performance across multiple frameworks. By unifying APIs, simplifying interoperability, and offering advanced graph optimization and container features, Ivy paves the way for a more flexible, modular, and efficient future of machine learning development. We now stand equipped to build and deploy models effortlessly across diverse environments, all using the same elegant Ivy codebase.

Check out the FULL CODES here. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post Ivy Framework Agnostic Machine Learning Build, Transpile, and Benchmark Across All Major Backends appeared first on MarkTechPost.

Meta AI has introduced Agents Research Environments (ARE), a modular simulation stack for creating and running agent tasks, and Gaia2, a follow-up benchmark to GAIA that evaluates agents in dynamic, write-enabled settings. ARE provides abstractions for apps, environments, events, notifications, and scenarios; Gaia2 runs on top of ARE and focuses on capabilities beyond search-and-execute.

https://ai.meta.com/research/publications/are-scaling-up-agent-environments-and-evaluations/

Why move from sequential to asynchronous interaction?

Most prior agent benchmarks pause the world while the model “thinks.” ARE decouples agent and environment time: the environment evolves while the agent is reasoning, injecting scheduled or stochastic events (e.g., replies, reminders, updates). This forces competencies like proactivity, interruption handling, and deadline awareness, which are under-measured in synchronous settings.

How is the ARE platform structured?

ARE is time-driven and treats “everything as an event.” Five core concepts organize simulations: Apps (stateful tool interfaces), Environments (collections of apps, rules, data), Events (logged happenings), Notifications (configurable observability to the agent), and Scenarios (initial state + scheduled events + verifier). Tools are typed as read or write, enabling precise verification of actions that mutate state. The initial environment, Mobile, mimics a smartphone with apps such as email, messaging, and calendar.

https://ai.meta.com/research/publications/are-scaling-up-agent-environments-and-evaluations/

What does Gaia2 actually measure?

Gaia2 targets general agent capabilities under realistic pressure: adaptability to environment responses, handling of ambiguity, noise robustness, time constraints (actions within tolerances), and Agent-to-Agent collaboration (coordinating sub-agents standing in for apps). Scenarios are verifiable and reproducible via deterministic seeds and oracle traces.

How large is the benchmark—800 or 1,120 scenarios?

The public dataset card specifies 800 scenarios across 10 universes. The paper’s experimental section references 1,120 verifiable, annotated scenarios in the Mobile environment (reflecting extended/augmented configurations used in the study). Practitioners will commonly encounter the 800-scenario release on Hugging Face, with the paper showing how the suite scales.

How are agents scored if the world is changing?

Gaia2 evaluates sequences of write actions against oracle actions with argument-level checks. Arguments are validated via hard (exact) or soft (LLM-judge) comparisons depending on type, maintaining causality and respecting relative-time constraints. This avoids the pitfall of judging only by end state when many trajectories are unsafe or policy-violating.

https://ai.meta.com/research/publications/are-scaling-up-agent-environments-and-evaluations/

Summary

ARE + Gaia2 shift the target from static correctness to correctness-under-change. If your agent claims to be production-ready, it should handle asynchrony, ambiguity, noise, timing, and multi-agent coordination—and do so with verifiable write-action traces. This release supplies: a controllable simulator, a challenging benchmark, and a transparent evaluation loop to stress real-world behaviors.

Check out the Paper, GitHub Codes and Technical Details.. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post Meta’s ARE + Gaia2 Set a New Bar for AI Agent Evaluation under Asynchronous, Event-Driven Conditions appeared first on MarkTechPost.

The convergence of artificial intelligence with physical systems marks a pivotal moment in technological evolution. Physical AI, where algorithms transcend digital boundaries to perceive, understand, and manipulate the tangible world, will fundamentally transform how enterprises operate across industries. These intelligent systems bridge the gap between digital intelligence and physical reality, unlocking unprecedented opportunities for efficiency and innovation. For many organizations, this opens the door to entirely new ways to delight their customers and, in turn, transform entire industries.
To accelerate this transformation, the AWS Generative AI Innovation Center, MassRobotics, and NVIDIA launched the Physical AI Fellowship, providing crucial support to startups developing next-generation robotics and automation solutions. We are pleased to be working with our first cohort fellows:

Bedrock Robotics – provides same-day hardware and software installation to provide autonomy to existing construction equipment fleets
Blue Water Autonomy – integrating hardware, software, and AI to enable uncrewed ships to operate on the open ocean for months at a time
Diligent Robotics – develop foundation models for autonomous humanoid robots in dynamic, human-facing environments
Generalist AI – developing end-to-end AI foundation models toward general-purpose robots, starting with a focus on dexterity
RobCo – offering modular hardware and a no-code system to automate tasks such as machine tending, palletizing, dispensing, or welding without upfront investment or specialist expertise
Tutor Intelligence – building AI-powered robots to help manufacturers and warehouses obtain immediate returns on investment
Wandercraft – developing exoskeletons to help with rehabilitation and restoring walking ability at home and in outpatient centers
Zordi – combining AI, robotics, and machine learning to innovate greenhouse agriculture

For businesses and public sector organizations, this convergence of AI and physical systems goes beyond incremental improvements, fundamentally rethinking what’s possible in their operations and customer experiences.
The Physical AI spectrum: from automation to true intelligence

As organizations evaluate their Physical AI initiatives, understanding where different solutions fall on the capability spectrum is crucial for strategic planning. Each level represents a distinct leap in autonomy and sophistication:

Level 1: Basic Physical Automation: This foundational stage involves systems that perform predefined tasks in tightly controlled environments. Think of industrial robots on assembly lines—highly efficient, but rigid and entirely dependent on human programming and oversight.
Level 2: Adaptive Physical Automation: At this stage, systems gain flexibility in task sequencing. While individual actions are still preprogrammed, they can adjust their order based on real-time environmental cues. Collaborative robots that change behavior when humans are nearby is a prime example.
Level 3: Partially Autonomous Physical AI: Here, systems demonstrate intelligent behavior, including planning, executing, and adapting tasks with limited human input. Robots that learn new processes through demonstration highlight this emerging autonomy.
Level 4: Fully Autonomous Physical AI: The most advanced level features systems capable of operating across varied domains with minimal supervision. These systems adapt fluidly to new scenarios and environmental changes. Although most commercial solutions remain at Levels 1 or 2, momentum toward full autonomy is accelerating.

Enabling technologies: the building blocks of Physical AI
The progression from basic automation to full autonomy requires sophisticated technological foundations. Several key innovations are driving this evolution:

Advanced control theory facilitates precise and reliable actuation.
High-fidelity perception models, powered by multimodal sensors, enable machines to interpret complex environments.
Edge AI accelerators support real-time inference at the point of action, crucial for latency-sensitive tasks.
Foundation models, trained on multimodal datasets, help provide generalizable intelligence across domains.
Digital twin systems play a pivotal role in enabling simulation, validation, and optimization of physical systems before real-world deployment, significantly accelerating development cycles.

Industry forces and investment momentum
Physical AI sits at the intersection of multiple high-growth industries, with the AI Robots sector alone projected to reach a staggering $124.26 billion by 2034. Alongside this, the closely related Digital Twin Technology industry is set to hit an even more impressive $379 billion in the same timeframe. These projections signal a fundamental shift in how enterprises approach automation, efficiency, and digital transformation.
Investors are keenly aware of this potential, focusing their attention on several key themes within the Physical AI space. Humanoid robotics has emerged as a particularly exciting frontier, with startups securing substantial funding rounds to develop general-purpose robotic workers capable of seamlessly operating in environments designed for humans. Simultaneously, there’s growing interest in foundation models for robotics – the development of sophisticated “robot brains” that can adapt to various tasks and control diverse robotic systems. This push towards more flexible, intelligent systems is complemented by continued investment in vertical-specific applications, where companies are leveraging Physical AI to address acute industry challenges, from streamlining warehouse logistics to revolutionizing agricultural practices. The breadth of Physical AI’s potential is further demonstrated by emerging applications in fields as diverse as surgical robotics, autonomous delivery systems, and advanced defense technologies. This expansion into new domains underscores the versatility and transformative power of Physical AI across sectors.
Real-world impact: quantifying the Physical AI transformation
While investment trends signal strong future potential, Physical AI is already delivering concrete results across industries. For example, Amazon’s supply chain has boosted efficiency by 25% through intelligent automation, while Foxconn cut manufacturing deployment times by 40%. In healthcare, AI-assisted procedures have led to 30% fewer complications and 25% shorter surgery durations, showcasing transformative outcomes.
According to a 2024 AI in manufacturing & energy report, 64% of manufacturers using AI in production already report positive ROI, with nearly one-third expecting returns of $2 to $5 for every dollar invested. These gains translate into efficiency improvements between 20-40%, cost savings of 15-30%, and the rise of innovative business models like Robot-as-a-Service.
In retail, digital twins are being used to explore the impact of different store layouts on shopper behavior and to test the integration of Physical AI with autonomous inventory management systems, helping retailers optimize their physical spaces and operations. Meanwhile, agriculture benefits from advancements in precision farming, crop monitoring, and automated harvesting—further highlighting Physical AI’s broad and growing impact.
The next frontier
The impact of Physical AI is already evident across industries, with organizations moving well beyond proofs-of-concept to delivering measurable business value. For participating cohorts, the Physical AI Fellowship will play a key role in helping innovative startups accelerate the path from research to commercial applications of this emerging technology. For enterprises of different sizes and sectors, successful integration of AI with physical systems will define industry leaders in the decade to come.
Learn more: 
Contact us to learn more about evaluating if your organization is set up to work as teammates, or if you’d like to dive deeper into skill development and risk posture for your physical AI plans.
Learn more about the Generative AI Innovation Center and how we provide expert tailored support from experimentation to production.

About the authors
Sri Elaprolu is a technology leader with over 25 years of experience spanning artificial intelligence, machine learning, and software engineering. As Director of the AWS Generative AI Innovation Center, Sri leads a global team of ML scientists and engineers applying the latest advances in generative AI to solve complex challenges for enterprises and the public sector.
Alla Simoneau is a technology and commercial leader with over 15 years of experience, currently serving as the Emerging Technology Physical AI Lead at Amazon Web Services (AWS), where she drives global innovation at the intersection of AI and real-world applications. With over a decade at Amazon, Alla is a recognized leader in strategy, team building, and operational excellence, specializing in turning cutting-edge technologies into real-world transformations for startups and enterprise customers.
Paul Amadeo is a seasoned technology leader with over 30 years of experience spanning artificial intelligence, machine learning, IoT systems, RF design, optics, semiconductor physics, and advanced engineering. As Technical Lead for Physical AI in the AWS Generative AI Innovation Center, Paul specializes in translating AI capabilities into tangible physical systems, guiding enterprise customers through complex implementations from concept to production. His diverse background includes architecting computer vision systems for edge environments, designing robotic smart card manufacturing technologies that have produced billions of devices globally, and leading cross-functional teams in both commercial and defense sectors. Paul holds an MS in Applied Physics from the University of California, San Diego, a BS in Applied Physics from Caltech, and holds six patents spanning optical systems, communication devices, and manufacturing technologies.
Randi Larson bridges the gap between AI innovation and executive strategy at the AWS Generative AI Innovation Center, shaping how organizations understand and translate technical breakthroughs into business value. She combines strategic storytelling with data-driven insight through global keynotes, Amazon’s first tech-for-good podcast, and conversations with industry and Amazon leaders on AI transformation. Before Amazon, Randi refined her analytical precision as a Bloomberg journalist and advisor to economic institutions, think tanks, and family offices on technology initiatives. Randi holds an MBA from Duke University’s Fuqua School of Business and a B.S. in Journalism and Spanish from Boston University.

In healthcare, the ability to quickly analyze and interpret medical reports is crucial for both healthcare providers and patients. While medical reports contain valuable information, they often remain underutilized due to their complex nature and the time-intensive process of analysis. This complexity manifests in several ways: the interpretation of multiple parameters and their relationships (such as various blood cell counts), the comparison of test results against standard reference ranges, and the need to analyze trends in health parameters over time. To address this challenge, we’ve conceptualized a medical reports analysis dashboard that illustrates how healthcare providers could enhance their interaction with medical data through a sample implementation
In this post, the created dashboard represents a convergent solution that brings together the power of Amazon Bedrock advanced AI capabilities, LangChain‘s document processing, and Streamlit‘s intuitive user interface. By using these technologies, we’ve created a system that not only stores and displays medical reports, but actively helps interpret them through natural language interactions and dynamic visualizations.
Solution overview
At the solution’s foundation are various large language models available through Amazon Bedrock, including Anthropic’s Claude series and Amazon Nova Foundation Models. You can select from options such as Claude Opus 4.1, Claude 3.7 Sonnet, Amazon Nova Pro, and others, each optimized for different performance and capability requirements. The chosen model processes natural language queries with medical context awareness, enabling detailed interpretation of healthcare data. With this flexibility, you can balance factors like accuracy, speed, and cost based on your specific needs. This is enhanced by LangChain’s document processing capabilities, which manage the retrieval system and maintain conversation context, facilitating accurate and relevant responses.
The solution’s data flow begins with medical reports securely stored in Amazon Simple Storage Service (Amazon S3), which are then processed through LangChain’s document handling system. When you interact with the Streamlit frontend, your queries are analyzed by Amazon Bedrock, while LangChain maintains the conversation context and manages document retrieval. The system processes this information and presents results through an intuitive interface featuring interactive visualizations.
These visualizations, powered by Plotly, include range comparison charts that clearly display normal versus actual values, bar charts for parameter comparisons, and trend lines for tracking changes over time. The Streamlit interface ties everything together, providing real-time interaction with the AI system while managing user session state and conversation history. This comprehensive approach helps ensure that medical professionals can quickly access, analyze, and interpret their medical reports through natural language queries while viewing supporting visual data.
The following is the architecture diagram of the solution that has four layers:

User Interface Layer: Streamlit Web App, Chat interface, Plotly data visualizations
Processing Layer: LangChain document processing, Conversation retrieval chain, Data parsing
AI/ML Layer: Amazon Bedrock, Amazon Bedrock embeddings, In-memory vector store
Storage Layer: Amazon S3 for medical reports, Conversation buffer memory

Prerequisites
Before deploying the Medical Reports Analysis Dashboard, you need:

An AWS account with Amazon Bedrock access enabled
AWS Identity and Access Management (IAM) permission for Amazon Bedrock and Amazon S3
AWS Command Line Interface (AWS CLI) installed and configured
An Amazon S3 bucket for storing medical reports in csv format

Follow Creating a general purpose bucket to create a bucket.
Sample reports provided are in the following repository. The command needed to upload reports is in the deployment section.

Python 3.9 or later with pip
Access to Amazon Bedrock Models. The solution supports multiple models including:

Anthropic’s Claude series (Opus 4.1, 3.7 Sonnet, Sonnet 4, and so on.)
Amazon Nova foundation model series (Nova Pro and Nova Lite)

We’ll be using a Python virtual environment (venv) for this project to provide a clean, isolated environment. Virtual environments help avoid package conflicts between projects and make dependency management more straightforward. While we’re using Python’s built-in venv, you could alternatively use miniconda or other environment managers.
Deployment
To get started with deployment, install the necessary packages on a local machine.

Clone the repository:

git clone https://github.com/aws-samples/sample-medical-analysis-dashboard.git

Navigate to the project directory.
Create and activate a virtual environment (recommended):

For Mac/Linux:

python3 -m venv venv
source venv/bin/activate

For Windows:

python3 -m venv venv
venvScriptsactivate

Update pip to the latest version:

python3 -m pip install –upgrade pip

Install required packages:

pip install -r requirements.txt

Project’s dependencies are listed in requirements.txt:

boto3
streamlit
unstructured
langchain-aws
langchain-community
pandas
plotly
numpy
docarray

These packages will handle AWS integration, web interface, data processing, and visualizations. They’ll be installed in our virtual environment during the deployment process. This setup helps ensure that the components are properly installed and isolated in a virtual environment for optimal performance.

Follow Configuring environment variables for the AWS CLI to configure AWS credentials.

export AWS_ACCESS_KEY_ID=’your-access-key’
export AWS_SECRET_ACCESS_KEY=’your-secret-key’

Upload sample CSV files to the S3 bucket created in prerequisites section:

Our repository contains two sample files:

basic_test.csv: Complete blood work with 15 parameters
blood_test.csv with basic parameters

The following is the content of basic_test.csv:

Parameter,Value,Reference_Range,Unit
Hemoglobin,13.8,13.5-17.5,g/dL
RBC,4.8,4.5-5.9,million/µL
WBC,8500,4000-11000,cells/µL
Glucose,92,70-100,mg/dL
Creatinine,1.0,0.7-1.3,mg/dL

Run the following commands to upload sample files to the S3 bucket:

aws s3 cp basic_test.csv s3://BUCKET_NAME/

aws s3 cp blood_test.csv s3://BUCKET_NAME/

Go to app.py line 68 and update the S3 bucket name in app.py to match your actual S3 bucket name.

BUCKET_NAME = “your-bucket-name”

Run the application:

streamlit run app.py

The dashboard will be available at http://localhost:8501. You can now interact with your medical reports through the web interface.
Using the dashboard
This section walks through the key features and demonstrates how to effectively use the dashboard for medical data analysis.
Dashboard interface overview
The following figures show the complete dashboard where the selected medical report is blood_test.csv from the repo showing the navigation pane and main content. The first figure also shows the first two graphs.

The following figure shows the second graph of the three that are included in this dashboard.

The dashboard interface is organized into three main sections for medical report analysis:

Document selection and model choice (navigation pane)

Selection of Amazon Bedrock model (for example: Claude Opus 4.1, Claude 3.7 Sonnet, or Amazon Nova Pro)
List of available medical reports in a dropdown menu
Currently analyzing blood_test.csv
Token usage display (input, output, and total tokens)

Chat analysis section

Clean chat interface for natural language queries
History of conversation maintained
Clear response formatting

Visualization area

Range comparison chart showing normal compared to actual values
Bar chart displaying the parameters
Trend lines for multiple parameters

Context-aware query system
The dashboard’s AI-powered query system demonstrates sophisticated understanding of medical reports through natural conversations. Here’s a sequence of interactions showing the system’s capabilities.
Question 1: Initial query about hemoglobin:

What is the hemoglobin level in report?

Question 2: Follow-up question demonstrating context awareness:

How does this compare to other parameters in the report? Are there any that stand out?

Question 3: Complex analysis request:

Can you analyze the distribution patterns of percentage-based measurements versus absolute values in this report, and identify any notable patterns in their reference ranges?

The system maintains conversation context while providing detailed insights from the medical reports, supporting responses with relevant data visualizations.
The solution can be further enhanced by fine-tuning the foundational model on organization-specific medical data, clinical questions, and domain expertise. This specialized training helps the model better understand medical terminology, standard protocols, and institution-specific practices. Additionally, organizations can use pre-trained medical LLMs available in AWS Marketplace, which are specifically optimized for healthcare use cases. When combined with the system’s existing capabilities, these specialized models can provide contextually relevant responses to medical queries while maintaining compliance with healthcare data governance requirements.
Amazon Bedrock guardrails should be configured to restrict the model from providing medical advice, prescriptions, or diagnoses, making sure responses are limited to data analysis and interpretation only.
Security considerations
While our current deployment uses dummy medical data for demonstration purposes, it’s crucial to consider security and compliance measures for real-world healthcare applications. Here are recommendations for enhancing security in a production environment:
Data privacy:

HIPAA compliance: Implement HIPAA-compliant practices, including access controls and audit trails.
Encryption: Use Amazon S3 server-side encryption (SSE-S3) for data at rest and TLS for data in transit.
Personally identifiable information (PII) protection:

Apply data masking for PII fields.
Control data access through role-based permissions.
Monitor model invocation using CloudWatch Logs and Amazon S3.
Configure Amazon Bedrock Guardrails. You can use guardrails to also restrict the model from providing medical advice, prescriptions, or diagnoses, limiting responses to data analysis and interpretation only.

Amazon S3 Configuration: Secure your medical data storage with the following S3 bucket settings

Enable versioning to maintain a complete audit trail and protect against accidental deletions or modifications
Block public access at both bucket and account levels
Implement strict bucket policies that limit access to specific IAM roles and enforce encryption in transit
Configure encryption (AES-256 or KMS) for all objects uploaded to the bucket

Recommended AWS security implementation:

IAM roles: Create specific IAM roles following the principle of least for each service
S3 bucket encryption: Enable default AES-256 encryption for all objects
Amazon Bedrock API access: Secure access using IAM roles and proper API key management
Audit logging: Activate AWS CloudTrail for comprehensive API call logging.

Log data access events on S3 buckets, Amazon Bedrock API calls, and IAM user and role activities
Monitor and record management events for S3 bucket configuration changes and policy updates

These are general recommendations. For a production healthcare application, consult with security experts and conduct a risk assessment to make sure all relevant compliance standards are met.
Clean up
To avoid ongoing AWS charges, follow these steps to clean up the resources created:

Delete the created Amazon S3 bucket
Delete the created local resources:

# Deactivate virtual environment
deactivate
# Remove project directory and virtual environment
rm -rf medical-analysis-dashboard/

Conclusion
In this post, we demonstrated the development of a conceptual Medical Reports Analysis Dashboard that combines Amazon Bedrock AI capabilities, LangChain’s document processing, and Streamlit’s interactive visualization features. The solution transforms complex medical data into accessible insights through a context-aware chat system powered by large language models available through Amazon Bedrock and dynamic visualizations of health parameters.
This project showcases how cloud and AI technologies can be applied to healthcare analytics, making medical report interpretation more intuitive and efficient. While our implementation uses dummy data for demonstration purposes, the architecture provides a foundation for building secure, compliance-aligned healthcare applications that can be enhanced to meet healthcare organizational requirements and security protocols.

About the authors
Aditya Ranjan is a Delivery Consultant with AWS, specializing in distributed systems architecture and cloud-native solutions. He collaborates with customers to design and implement well-architected technical solutions using AWS’s latest technologies, including generative AI services, enabling them to achieve their business goals and objectives.
Shubham Tiwari is a Solutions Architect at AWS specializing in Modernisation, containers and Security. He has been helping customers in deploying highly scalable, resilient and cost optimised architecture on AWS.

This post was written with Ajay Nyamati from Kitsa.
The clinical trial industry conducts medical research studies to evaluate the safety, efficacy, and effectiveness of new drugs, treatments, or medical devices before they reach the market. The industry is a cornerstone of medical innovation, yet it continues to face a fundamental bottleneck: selection of the right trial sites based on the requirement by clinical trial sponsors and contract research organizations (CROs).
Although there are tens of thousands of potential research sites worldwide, the decision-making process is still heavily influenced by personal networks, limited visibility, and incomplete data. The result is delayed trial launches, underutilized site capacity, and missed opportunities for both sponsors and research centers.
Key challenges in site selection include:

Data fragmentation: Site performance and operational data are scattered across siloed systems, inconsistent formats, and unstructured online sources.
Manual effort and low coverage: Sponsors and CROs often review only a fraction of the available sites due to the time and cost of manual analysis.
Over-reliance of Key Opinion Leaders (KOLs): Personal preference and relationships often outweigh objective performance metrics.
Missed opportunities for capable sites: Many high-quality sites are overlooked because they lack a centralized platform to showcase their capabilities.
Knowledge hoarding: Organizations with large datasets often keep them proprietary, limiting industry-wide progress.

In this post, we’ll show how Kitsa used Amazon Quick Automate to transform their clinical trial site selection solution. Amazon Quick Automate, a capability of Amazon Quick Suite, enables enterprises to build, deploy and maintain resilient workflow automations at scale. Amazon Quick Suite helps business users make better decisions faster and act on them by unifying AI agents for research, business insights, and automation into a single experience.
Kitsa, a health-tech company specializing in AI-driven clinical trial recruitment and site selection, is tackling the challenge in site selection. By combining demographic data, disease prevalence insights, historical trial performance, and operational site metrics, Kitsa has developed an agentic analytics engine that matches sponsors with the most suitable sites for their studies. This approach requires consolidating and analyzing data from hundreds of fragmented sources, including websites of clinical trial sites, clinical trial registries, investigator resumes, regulatory filings, publications, and conference abstracts. Traditionally, this has been a slow, manual process that pushed trial start dates by months.
To address this, Kitsa turned to Amazon Web Services (AWS) to build a scalable, secure, and compliant automation pipeline that unifies this data into a single decision-making engine. Using Quick Automate, a generative AI–powered workflow automation capability of Amazon Quick Suite, Kitsa can rapidly extract, normalize, and analyze site data at scale. With an advanced multi-agent automation architecture engineered for enterprise-scale deployment, Quick Automate combines UI automation, API integrations, and workflow orchestration in a single, fully managed solution.
Quick Automate uses generative AI to analyze inputs from the user and suggests a workflow that can be modified and extended to take action across business systems and UIs, engaging a human when needed. Through specialized AI agents, Quick Automate helps organizations to automate complex processes across applications and departments. It also reduces operational costs through usage-based pricing.
By using AWS services, Kitsa is transforming site selection from a slow, relationship-driven process into a fast, data-driven, and globally scalable system.
Solution overview and details
Kitsa required a process automation solution capable of navigating websites, extracting over 50 distinct data points, and compiling the results in a structured format. The solution needed to be highly reliable, scalable to hundreds of thousands of websites, and accurate. Given that Kitsa operates in the life sciences and healthcare sector, which is heavily regulated, they also needed a secure, compliant solution that meets the industry’s strict standards.
The automation was built using Quick Automate, designed for enterprise-scale workflow automation. A key component of the solution is a state-of-the-art UI Agent, configured to autonomously perform website navigation and data extraction. The UI Agent is part of Quick Automate, enabling complex browser-based workflows.
The UI Agent takes natural language input and produces structured outputs—essential for reliably capturing more than 50 data points from each website. It was configured to extract information efficiently and consistently, maintaining both accuracy and compliance. The AWS team collaborated closely with the Kitsa team to design and refine specialized prompts, helping the automation perform optimally for the customer’s needs. The following architecture diagram illustrates the workflow.

Workflow architecture and implementation
The automation workflow uses the following:
Case initialization and parallel processing
The automation begins by fetching cases, where each case contains the URL that needs information extraction. The case management functionality enables parallelization of website processing and evaluation, reducing processing time through concurrent execution of multiple cases.
Intelligent data extraction
For each case, the UI Agent navigates to the specified URL and extracts required information while applying AI reasoning concerning the content. The information extraction process utilizes natural language instructions provided to the UI Agent task. It then delivers results in a structured output format, so downstream workflow steps can consume them without extra parsing.
Human-in-the-loop integration
When website information extraction shows lower confidence, the system can automatically route cases to human reviewers for manual assessment. This human-in-the-loop (HILO) approach maintains quality control while allowing automated processing.
Data persistence and storage
Processed cases are systematically saved and written to an Excel spreadsheet within the workflow. The completed files are then uploaded to an Amazon Simple Storage Service (Amazon S3) bucket through integrated S3 connectors, providing secure and accessible data storage.
Robust exception handling
The workflow incorporates exception handling mechanisms to gracefully manage scenarios where websites are not found, under construction, or otherwise inaccessible. The workflow returns accurate error messages and continues processing subsequent websites without interrupting the overall workflow execution, resulting in operational continuity and reliability.
Results
With Quick Automate powering the Kitsa large-scale data extraction and integration workflow solution, the impact was immediate and measurable:

91% cost savings: Compared to the legacy manual process it lowered operational expenses while dramatically expanding the number of sites analyzed.
96% faster data acquisition: Kitsa is able to process in days what previously took months, accelerating the entire site feasibility process.
96% coverage in data extraction: Surpasses manual review while maintaining consistency across hundreds of thousands of processed websites.
Full regulatory compliance: Meets all data security, privacy, and auditability standards required in life sciences and healthcare.

The solution now directly powers the Kitsa Site Finder Agent, which evaluates hundreds of site-specific parameters (from past recruitment speed to infrastructure readiness), and ranks them with a trial-specific algorithm. Sponsors can now compare sites on hard evidence rather than subjective impressions, and eligible sites can showcase their capabilities to pharma companies for the first time in a structured, data-rich format.
As Rohit Banga, Co-Founder & CTO of Kitsa, explains:

“With Amazon Quick Automate, we were able to break through one of the biggest bottlenecks in site selection — collecting and unifying high-quality data at scale. This allowed our Site Finder Agent to evaluate more sites, more fairly, and with more precision than ever before. Our results show 96% coverage in data extraction, 91% cost savings compared to legacy manual processes, and 96% faster data acquisition – processing in days what previously took months.”

Conclusion
Clinical trial site selection has long been a critical bottleneck in medical research, with fragmented data and manual processes causing significant delays and missed opportunities. Kitsa addressed this challenge by using the Automate capability of Amazon Quick Suite in their automated site selection solution.
With the solution Kitsa can automatically extract and analyze over 50 distinct data points from hundreds of thousands of websites. They are achieving remarkable results with 96% coverage in data extraction and 91% cost savings compared to manual processes. Kitsa also reduced their data acquisition time by 96% while maintaining full regulatory compliance in the heavily regulated healthcare sector.
Their Site Finder Agent now evaluates hundreds of site-specific parameters objectively, helping pharmaceutical companies to make evidence-based decisions and allowing trial sites to showcase their capabilities in a structured format. This transformation demonstrates how Quick Automate can solve complex industry challenges while significantly improving efficiency, accuracy, and fairness in clinical trial site selection.
Contact an AWS Representative to know how we can help accelerate your business.

About the authors
Chethan Shriyan is a Principal Product Manager – Technical at AWS. He has 12+ years of experience in product and business management. Chethan is passionate about building and delivering technology products that create meaningful impact in customers’ lives.
Ajay Nyamati is the co-founder and CEO of Kitsa – a healthtech company using AI and data automation to transform clinical trials. With 20+ years of Sales & Strategy in global companies, Ajay has spent 10+ years in the Digital Health space across payors, providers and pharma. Before co-founding Kitsa, he was the business leader for clinical trials solutions in Amazon Web Services.
Reagan Rosario brings over a decade of technical expertise to his role as a Sr. Specialist Solutions Architect in Generative AI at AWS. Reagan transforms enterprise systems through strategic implementation of AI-powered cloud solutions, automated workflows, and innovative architecture design. His specialty lies in guiding organizations through digital evolution—preserving core business value while implementing cutting-edge generative AI capabilities that dramatically enhance operations and create new possibilities.

Google AI Research team has brought a production shift in Voice Search by introducing Speech-to-Retrieval (S2R). S2R maps a spoken query directly to an embedding and retrieves information without first converting speech to text. The Google team positions S2R as an architectural and philosophical change that targets error propagation in the classic cascade modeling approach and focuses the system on retrieval intent rather than transcript fidelity. Google research team states Voice Search is now powered by S2R.

https://research.google/blog/speech-to-retrieval-s2r-a-new-approach-to-voice-search/

From cascade modeling to intent-aligned retrieval

In the traditional cascade modeling approach, automatic speech recognition (ASR) first produces a single text string, which is then passed to retrieval. Small transcription errors can change query meaning and yield incorrect results. S2R reframes the problem around the question “What information is being sought?” and bypasses the fragile intermediate transcript.

Evaluating the potential of S2R

Google’s research team analyzed the disconnect between word error rate (WER) (ASR quality) and mean reciprocal rank (MRR) (retrieval quality). Using human-verified transcripts to simulate a cascade groundtruth “perfect ASR” condition, the team compared (i) Cascade ASR (real-world baseline) vs (ii) Cascade groundtruth (upper bound) and observed that lower WER does not reliably predict higher MRR across languages. The persistent MRR gap between the baseline and groundtruth indicates room for models that optimize retrieval intent directly from audio.

https://research.google/blog/speech-to-retrieval-s2r-a-new-approach-to-voice-search/

Architecture: dual-encoder with joint training

At the core of S2R is a dual-encoder architecture. An audio encoder converts the spoken query into a rich audio embedding that captures semantic meaning, while a document encoder generates a corresponding vector representation for documents. The system is trained with paired (audio query, relevant document) data so that the vector for an audio query is geometrically close to vectors of its corresponding documents in the representation space. This training objective directly aligns speech with retrieval targets and removes the brittle dependency on exact word sequences.

Serving path: streaming audio, similarity search, and ranking

At inference time, the audio is streamed to the pre-trained audio encoder to produce a query vector. This vector is used to efficiently identify a highly relevant set of candidate results from Google’s index; the search ranking system—which integrates hundreds of signals—then computes the final order. The implementation preserves the mature ranking stack while replacing the query representation with a speech-semantic embedding.

Evaluating S2R on SVQ

On the Simple Voice Questions (SVQ) evaluation, the post presents a comparison of three systems: Cascade ASR (blue), Cascade groundtruth (green), and S2R (orange). The S2R bar significantly outperforms the baseline Cascade ASR and approaches the upper bound set by Cascade groundtruth on MRR, with a remaining gap that the authors note as future research headroom.

Open resources: SVQ and the Massive Sound Embedding Benchmark (MSEB)

To support community progress, Google open-sourced Simple Voice Questions (SVQ) on Hugging Face: short audio questions recorded in 26 locales across 17 languages and under multiple audio conditions (clean, background speech noise, traffic noise, media noise). The dataset is released as an undivided evaluation set and is licensed CC-BY-4.0. SVQ is part of the Massive Sound Embedding Benchmark (MSEB), an open framework for assessing sound embedding methods across tasks.

Key Takeaways

Google has moved Voice Search to Speech-to-Retrieval (S2R), mapping spoken queries to embeddings and skipping transcription.

Dual-encoder design (audio encoder + document encoder) aligns audio/query vectors with document embeddings for direct semantic retrieval.

In evaluations, S2R outperforms the production ASR→retrieval cascade and approaches the ground-truth transcript upper bound on MRR.

S2R is live in production and serving multiple languages, integrated with Google’s existing ranking stack.

Google released Simple Voice Questions (SVQ) (17 languages, 26 locales) under MSEB to standardize speech-retrieval benchmarking.

Editorial Comments

Speech-to-Retrieval (S2R) is a meaningful architectural correction rather than a cosmetic upgrade: by replacing the ASR→text hinge with a speech-native embedding interface, Google aligns the optimization target with retrieval quality and removes a major source of cascade error. The production rollout and multilingual coverage matter, but the interesting work now is operational—calibrating audio-derived relevance scores, stress-testing code-switching and noisy conditions, and quantifying privacy trade-offs as voice embeddings become query keys.

Check out the Technical details here. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post Google Introduces Speech-to-Retrieval (S2R) Approach that Maps a Spoken Query Directly to an Embedding and Retrieves Information without First Converting Speech to Text appeared first on MarkTechPost.

As AI agents evolve beyond simple chatbots, new design patterns have emerged to make them more capable, adaptable, and intelligent. These agentic design patterns define how agents think, act, and collaborate to solve complex problems in real-world settings. Whether it’s reasoning through tasks, writing and executing code, connecting to external tools, or even reflecting on their own outputs, each pattern represents a distinct approach to building smarter, more autonomous systems. Here are five of the most popular agentic design patterns every AI engineer should know.

ReAct Agent

A ReAct agent is an AI agent built on the “reasoning and acting” (ReAct) framework, which combines step-by-step thinking with the ability to use external tools. Instead of following fixed rules, it thinks through problems, takes actions like searching or running code, observes the results, and then decides what to do next.

The ReAct framework works much like how humans solve problems — by thinking, acting, and adjusting along the way. For example, imagine planning dinner: you start by thinking, “What do I have at home?” (reasoning), then check your fridge (action). Seeing only vegetables (observation), you adjust your plan — “I’ll make pasta with vegetables.” In the same way, ReAct agents alternate between thoughts, actions, and observations to handle complex tasks and make better decisions.

The image below illustrates the basic architecture of a ReAct Agent. The agent has access to various tools that it can use when required. It can independently reason, decide whether to invoke a tool, and re-run actions after making adjustments based on new observations. The dotted lines represent conditional paths—showing that the agent may choose to use a tool node only when it deems it necessary.

CodeAct Agent

A CodeAct Agent is an AI system designed to write, run, and refine code based on natural language instructions. Instead of just generating text, it can actually execute code, analyze the results, and adjust its approach — allowing it to solve complex, multi-step problems efficiently.

At its core, CodeAct enables an AI assistant to:

Generate code from natural language input

Execute that code in a safe, controlled environment

Review the execution results

Improve its response based on what it learns

The framework includes key components like a code execution environment, workflow definition, prompt engineering, and memory management, all working together to ensure the agent can perform real tasks reliably.

A good example is Manus AI, which uses a structured agent loop to process tasks step by step. It first analyzes the user’s request, selects the right tools or APIs, executes commands in a secure Linux sandbox, and iterates based on feedback until the job is done. Finally, it submits results to the user and enters standby mode, waiting for the next instruction.

Self-Reflection

A Reflection Agent is an AI that can step back and evaluate its own work, identify mistakes, and improve through trial and error—similar to how humans learn from feedback.

This type of agent operates in a cyclical process: it first generates an initial output, such as text or code, based on a user’s prompt. Next, it reflects on that output, spotting errors, inconsistencies, or areas for improvement, often applying expert-like reasoning. Finally, it refines the output by incorporating its own feedback, repeating this cycle until the result reaches a high-quality standard.

Reflection Agents are especially useful for tasks that benefit from self-evaluation and iterative improvement, making them more reliable and adaptable than agents that generate content in a single pass.

Multi-Agent Workflow

A Multi-Agent System uses a team of specialized agents instead of relying on a single agent to handle everything. Each agent focuses on a specific task, leveraging its strengths to achieve better overall results.

This approach offers several advantages: focused agents are more likely to succeed on their specific tasks than a single agent managing many tools; separate prompts and instructions can be tailored for each agent, even allowing the use of fine-tuned LLMs; and each agent can be evaluated and improved independently without affecting the broader system. By dividing complex problems into smaller, manageable units, multi-agent designs make large workflows more efficient, flexible, and reliable.

The above image visualizes a Multi-Agent System (MAS), illustrating how a single user prompt is decomposed into specialized tasks handled in parallel by three distinct agents (Research, Coding, and Reviewer) before being synthesized into a final, high-quality output.

Agentic RAG

Agentic RAG agents take information retrieval a step further by actively searching for relevant data, evaluating it, generating well-informed responses, and remembering what they’ve learned for future use. Unlike traditional Native RAG, which relies on static retrieval and generation processes, Agentic RAG employs autonomous agents to dynamically manage and improve both retrieval and generation. 

The architecture consists of three main components. 

The Retrieval System fetches relevant information from a knowledge base using techniques like indexing, query processing, and algorithms such as BM25 or dense embeddings. 

The Generation Model, typically a fine-tuned LLM, converts the retrieved data into contextual embeddings, focuses on key information using attention mechanisms, and generates coherent, fluent responses. 

The Agent Layer coordinates the retrieval and generation steps, making the process dynamic and context-aware while enabling the agent to remember and leverage past information. 

Together, these components allow Agentic RAG to deliver smarter, more contextual answers than traditional RAG systems.

The post 5 Most Popular Agentic AI Design Patterns Every AI Engineer Should Know appeared first on MarkTechPost.

A significant development is set to transform AI in healthcare. Researchers at Stanford University, in collaboration with ETH Zurich and tech leaders including Google Research and Amazon, have introduced OpenTSLM, a novel family of Time-Series Language Models (TSLMs).

This breakthrough addresses a critical limitation in current LLMs by enabling them to interpret and reason over complex, continuous medical time-series data, such as ECGs, EEGs, and wearable sensor streams, a feat where even frontier models like GPT-4o have struggled.

The Critical Blind Spot: LLM Limitations in Time-Series Analysis

Medicine is fundamentally temporal. Accurate diagnosis relies heavily on tracking how vital signs, biomarkers, and complex signals evolve. Despite the proliferation of digital health technology, today’s most advanced AI models have struggled to process this raw, continuous data.

The core challenge lies in the “modality gap”, the difference between continuous signals (like a heartbeat) and the discrete text tokens that LLMs understand. Previous attempts to bridge this gap by converting signals into text have proven inefficient and difficult to scale.

Why Vision-Language Models (VLMs) Fail at Time-Series Data

A common workaround has been to convert time-series data into static images (line plots) and input them into advanced Vision-Language Models (VLMs). However, the OpenTSLM research demonstrates this approach is surprisingly ineffective for precise medical data analysis.

VLMs are primarily trained on natural photographs; they recognize objects and scenes, not the dense, sequential dynamics of data visualizations. When high-frequency signals like an ECG are rendered into pixels, crucial fine-grained information is lost. Subtle temporal dependencies and high-frequency changes, vital for identifying heart arrhythmias or specific sleep stages, become obscured.

The study confirms that VLMs struggle significantly when analyzing these plots, highlighting that time series must be treated as a distinct data modality, not merely a picture.

Introducing OpenTSLM: A Native Modality Approach

OpenTSLM integrates time series as a native modality directly into pretrained LLMs (such as Llama and Gemma), enabling natural language querying and reasoning over complex health data. 

https://www.arxiv.org/abs/2510.02410

The research team explored two distinct architectures:

Architecture Deep Dive: SoftPrompt vs. Flamingo

1. OpenTSLM-SoftPrompt (Implicit Modeling)

This approach encodes time-series data into learnable tokens, which are then combined with text tokens (soft prompting). While efficient for short data bursts, this method scales poorly. Longer sequences require exponentially more memory, making it impractical for comprehensive analysis.

https://www.arxiv.org/abs/2510.02410

2. OpenTSLM-Flamingo (Explicit Modeling)

Inspired by the Flamingo architecture, this is the breakthrough solution for scalability. It explicitly models time series as a separate modality. It uses a specialized encoder and a Perceiver Resampler to create a fixed-size representation of the data, regardless of its length, and fuses it with text using gated cross-attention.

https://www.arxiv.org/abs/2510.02410

OpenTSLM-Flamingo maintains stable memory requirements even with extensive data streams. For instance, during training on complex ECG data analysis, the Flamingo variant required only 40 GB of VRAM, compared to 110 GB for the SoftPrompt variant using the same LLM backbone.

Performance Breakthroughs: Outperforming GPT-4o

The results demonstrate the clear superiority of the specialized TSLM approach. To benchmark performance, the team created three new Chain-of-Thought (CoT) datasets focused on medical reasoning: HAR-CoT (activity recognition), Sleep-CoT (EEG sleep staging), and ECG-QA-CoT (ECG question answering).

Sleep Staging: OpenTSLM achieved a 69.9% F1 score, vastly outperforming the best fine-tuned text-only baseline (9.05%).

Activity Recognition: OpenTSLM reached a 65.4% F1 score

Here is an example of human activity recognition COT.

https://www.arxiv.org/abs/2510.02410

Here is an example of Sleep activity detection:

https://www.arxiv.org/abs/2510.02410

Remarkably, even small-scale OpenTSLM models (1 billion parameters) significantly surpassed GPT-4o. Whether processing the data as text tokens (where GPT-4o scored only 15.47% on Sleep-CoT) or as images, the frontier model failed to match the specialized TSLMs.

This finding underscores that specialized, domain-adapted AI architectures can achieve superior results without massive scale, paving the way for efficient, on-device medical AI deployment.

Clinical Validation at Stanford Hospital: Ensuring Trust and Transparency

A crucial element of Medical AI is trust. Unlike traditional models that output a single classification, OpenTSLM generates human-readable rationales (Chain-of-Thought), explaining its predictions. This AI transparency is vital for clinical settings.

To validate the quality of this reasoning, an expert review was conducted with five cardiologists from Stanford Hospital. They assessed the rationales generated by the OpenTSLM-Flamingo model for ECG interpretation.

The evaluation found that the model provided a correct or partially correct ECG interpretation in an impressive 92.9% of cases. The model showed exceptional strength in integrating clinical context (85.1% positive assessments), demonstrating sophisticated reasoning capabilities over raw sensor data.

The Future of Multimodal Machine Learning

The introduction of OpenTSLM marks a significant advancement in multimodal machine learning. By effectively bridging the gap between LLMs and time-series data, this research lays the foundation for general-purpose TSLMs capable of handling diverse longitudinal data, not just in healthcare, but also in finance, industrial monitoring, and beyond.

To accelerate innovation in the field, the Stanford and ETH Zurich teams have open-sourced all code, datasets, and trained model weights.

Check out the Paper here. Feel free to check out our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
The post Meet OpenTSLM: A Family of Time-Series Language Models (TSLMs) Revolutionizing Medical Time-Series Analysis appeared first on MarkTechPost.