Why Python Decorator Runtimes Score Low on Machine Discovery but Win at Runtime

Correction (2026-10-12): This post originally said E2B scored 31/100 on front-door discovery in our Sandboxes Edition 1 study. In that study E2B scored 75/100 and Modal 36/100; the 31 comes from State of Agent Readiness 2026, a different scanner and scale. It also said Modal and E2B "completely dominated" the runtime benchmark and attributed E2B's token count to its response envelopes. The runtime token, step and time figures were self-reported by the agent, prompts differed by provider (Modal's prompts spelled out the exact SDK calls), Modal's Trial 6 ran no tests, and the cause of the token differences was not measured. Full list with evidence: Sandboxes Edition 1 errata (also in the research repository).

In the emerging field of Agent Readiness Standards (ARS), developer tooling is increasingly analyzed by how effectively autonomous AI coding agents can discover, parse, and invoke APIs.

When automated machine parsers audit developer surfaces, they look for standard machine-readable artifacts: clean OpenAPI 3.1 specifications, well-structured /llms.txt directories, JSON schemas, and REST endpoint catalogs.

By these traditional static metrics, Python decorator runtimes and code-first infrastructure platforms—Modal is the clearest example—can receive low front-door discovery scores:

• They rarely expose extensive public REST endpoints.
• Their documentation sites frequently lack exhaustive OpenAPI JSON schemas.
• Their execution model relies heavily on language-level abstractions rather than generic HTTP webhooks.

In our recent investigation, The Glintbase Integration Index: Cloud Sandboxes for Autonomous AI Coding Agents, Modal received a Front-Door Agent Readiness Score of just 36 out of 100 (Grade D), the lowest in the cohort, because its site exposes no OpenAPI or machine-readable REST surface. (Corrected 2026-10-12: this sentence previously gave E2B a score of 31. E2B scored 75/100 (Grade B) in this study; 31 is its score in State of Agent Readiness 2026, which uses a different scanner and scale.)

Yet when we deployed autonomous coding agents into live execution trials, Modal ran efficiently: it had the lowest self-reported median command time (19.5s) and the fewest transcript-measured model turns (median 12.5) in the cohort, while E2B had the lowest self-reported median token count (1,865). Two caveats: every Modal prompt spelled out the exact SDK calls, which helps explain the low turn count, and Modal's bottle test-suite trial ran no tests, so it passed 5 of 6 trials. (Corrected 2026-10-12: previously described as Modal and E2B "completely dominating" the runtime benchmark.)

What explains this massive divergence between machine discovery and runtime reality?

---

1. The Limitation of Document-Level Machine Discovery

Static discovery benchmarks evaluate developer documentation using a document-retrieval mindset. They assess whether a web crawler or LLM scraper can find an endpoint definition, read an authentication guide, and construct an HTTP curl command.

Platforms designed around REST APIs (like traditional cloud providers) fit this discovery model neatly:

• Every capability maps to an HTTP verb and URL path (POST /v1/sandboxes).
• Input payloads are defined by JSON Schema.
• Authentication is a standard Bearer token in an HTTP header.

However, constructing and executing raw HTTP requests from inside an autonomous coding agent is inherently fragile:

• The agent must correctly serialize complex JSON parameters.
• It must handle HTTP status codes, redirects, and parsing errors manually.
• It must maintain session state and persistent WebSocket connections across multiple tool turns.

---

2. Why Language-Level Abstractions Beat REST for AI Agents

Python decorator runtimes bypass the REST translation layer entirely by exposing typed, idiomatic language abstractions:

A. The LLM Writes Idiomatic Code, Not Network Calls

Modern frontier models (Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Pro) are trained on billions of lines of high-quality Python code. They understand Python classes, context managers, and type hints with extraordinary fidelity. When an agent interacts with a typed SDK like Modal (modal.Sandbox.create) or E2B (from e2b import Sandbox), the model generates idiomatic code naturally. It does not have to invent HTTP headers or debug JSON serialization errors.

B. Python Type Hints Enforce Correctness at Generation Time

Typed SDKs provide immediate compiler-level and runtime feedback:

• If a method requires an explicit parameter, Python raises an immediate TypeError that clearly communicates the missing argument.
• Conversely, a REST API often returns a generic HTTP 400 Bad Request with an ambiguous error string that leaves the agent guessing which key in a nested JSON payload was malformed.

C. Bounded Execution Envelopes Minimize Context Inflation

When an agent communicates via a dedicated Python client library, the library wraps responses in clean, structured dataclasses:

• process.stdout returns clean string content.
• process.exit_code returns an integer.
• The agent does not have to strip HTTP headers, chunked transfer encodings, or TLS handshake logs from its context window.

That is the design argument. In our benchmark E2B reported the lowest median token count (1,865 tokens), but those counts were self-reported by the agent, and we did not measure how much of any difference came from response envelopes. (Corrected 2026-10-12: previously presented as the measured reason for E2B's token count.)

---

3. The Front-Door vs. Runtime Dilemma for Platform Architects

This divergence exposes a critical dilemma for infrastructure companies building developer platforms in the age of AI:

1. If you only optimize for static discovery: You might achieve a high score on automated documentation linters by publishing thousands of markdown pages and OpenAPI files, yet your autonomous agents may still struggle at runtime due to network protocol friction or out-of-band setup such as billing gates.
2. If you only optimize for SDK ergonomics: Your runtime execution will be lightning fast, but new autonomous agents using web search or zero-shot retrieval might struggle to discover your API if your documentation lacks structured machine entrypoints like /llms.txt.

---

The Solution: The Dual-Surface Architecture

To build infrastructure that thrives both at the front door and at runtime, platforms must adopt a Dual-Surface Architecture:

1. At the Front Door (Discovery Layer):
   • Publish a machine-readable /llms.txt file pointing directly to idiomatic code examples.
   • Provide minimal, copy-pasteable zero-hop Python quickstarts.
   • Expose OpenAPI 3.1 specifications for control-plane endpoints.
2. At Runtime (Execution Layer):
   • Provide first-class, typed Python and TypeScript SDKs.
   • Return clean, structured execution envelopes that protect LLM context windows.
   • Make teardown deterministic via context managers, and make leak checks fail loudly rather than cleaning up silently.

When infrastructure vendors align both surfaces, they eliminate the Integration Tax and unlock the full potential of autonomous AI coding agents.

Read the complete empirical study in The Glintbase Integration Index (Sandboxes Edition 1).