Operating the Dify Agent Run Server

This guide describes how to run the MVP Dify Agent API server. The server is implemented in dify-agent/src/dify_agent/server/app.py and uses Redis for run records and per-run event streams. Optional E2B usage collection is triggered by the API's existing Celery Beat schedule through a separate one-shot endpoint. It adds no operation logging, polling loop, leader election or accounting context to business runtime execution.

Default local startup

Start Redis, then run one FastAPI/uvicorn process:

uv run --project dify-agent uvicorn dify_agent.server.app:app --reload

By default, the FastAPI lifespan creates:

• one Redis-backed run store used by HTTP routes
• shared plugin-daemon and Dify API inner httpx.AsyncClient instances
• one deployment-selected, stateless runtime backend profile when configured
• one process-local scheduler that starts background asyncio run tasks

This means local development needs one uvicorn process plus Redis, and plugin-backed runs also need a reachable Dify plugin daemon. Run execution still happens outside request handlers, so client disconnects do not cancel the agent run.

Configuration

ServerSettings loads environment variables with the DIFY_AGENT_ prefix. It also reads .env and dify-agent/.env when present.

OpenShell-specific settings are listed in the OpenShell configuration reference.

Independent E2B execution accounting, activation and rollout checks are described in E2B runtime metering.

Environment variable	Default	Description
DIFY_AGENT_REDIS_URL	redis://localhost:6379/0	Redis connection URL.
DIFY_AGENT_REDIS_PREFIX	dify-agent	Prefix for Redis record and event keys.
DIFY_AGENT_SHUTDOWN_GRACE_SECONDS	30	Seconds to wait for active local runs during graceful shutdown before cancellation.
DIFY_AGENT_RUN_RETENTION_SECONDS	7200	Seconds to retain Redis run records and per-run event streams after their last write; defaults to 2 hours.
DIFY_AGENT_RUN_EVENT_STREAM_MAX_LENGTH	5000	Approximate target maximum for replayable events retained in each per-run Redis Stream.
DIFY_AGENT_STREAM_TEXT_DELTA_COALESCING_ENABLED	true	Set false to publish each text delta without coalescing.
DIFY_AGENT_STREAM_TEXT_DELTA_FLUSH_INTERVAL_MS	100	Soft debounce interval for compatible text deltas. Already-ready source events may continue to merge after this interval; a waiting source triggers the timed flush. Must be greater than zero; use DIFY_AGENT_STREAM_TEXT_DELTA_COALESCING_ENABLED=false to disable coalescing.
DIFY_AGENT_STREAM_TEXT_DELTA_MAX_CHARS	4096	Character threshold that flushes a buffered text-delta event immediately.
DIFY_AGENT_RUN_TIMEOUT_SECONDS	3600	Wall-clock deadline in seconds for the Pydantic AI agent.run(...) model/tool loop. Deadline failures use agent_run_limit_exceeded. Its default intentionally matches DIFY_AGENT_E2B_ACTIVE_TIMEOUT_SECONDS, but the settings are independently configurable.
DIFY_AGENT_BINDING_FILE_DOWNLOAD_COMMAND_TIMEOUT_SECONDS	210	Shell command deadline for running the sandbox dify-agent file upload --no-download-link conversion. Keep it above the CLI's 180-second upload deadline.
DIFY_AGENT_API_TOKEN	empty	Optional Bearer token required by private run, Execution Binding, Home Snapshot, and Binding file control-plane routes. Must match Dify API AGENT_BACKEND_API_TOKEN.
DIFY_AGENT_PLUGIN_DAEMON_URL	http://localhost:5002	Base URL for the Dify plugin daemon.
DIFY_AGENT_PLUGIN_DAEMON_API_KEY	empty	API key sent to the Dify plugin daemon.
DIFY_AGENT_INNER_API_URL	http://localhost:5001	Dify API service root used when dify-agent calls /inner/api/... endpoints.
DIFY_AGENT_INNER_API_KEY	empty	API key sent to Dify API inner plugin endpoints. Set this to Dify API INNER_API_KEY_FOR_PLUGIN (Docker: PLUGIN_DIFY_INNER_API_KEY).
DIFY_AGENT_RUNTIME_BACKEND	local	Selects one coherent local, enterprise, e2b, or openshell Home Snapshot + Execution Binding backend profile.
DIFY_AGENT_LOCAL_SANDBOX_ENDPOINT	empty	Local shellctl data-plane URL. With the default Local selection, leaving it empty disables dify.runtime and resource endpoints.
DIFY_AGENT_LOCAL_SANDBOX_AUTH_TOKEN	empty	Required nonblank bearer token when the Local endpoint is configured; must match shellctl's SHELLCTL_AUTH_TOKEN.
DIFY_AGENT_LOCAL_SANDBOX_MATERIALIZED_HOME_ROOT	/home/dify	Root directory, on the Local shellctl filesystem, for per-Binding materialized Homes.
DIFY_AGENT_LOCAL_SANDBOX_WORKSPACE_ROOT	/workspace	Root directory, on the Local shellctl filesystem, for mutable Workspaces.
DIFY_AGENT_LOCAL_SANDBOX_HOME_SNAPSHOT_ROOT	/home/dify/.snapshots	Root directory, on the Local shellctl filesystem, for immutable Home Snapshots.
DIFY_AGENT_ENTERPRISE_SANDBOX_GATEWAY_ENDPOINT	empty	Enterprise Gateway endpoint required by configuration. Default-Home Bindings are supported; immutable Home Snapshot operations remain unsupported.
DIFY_AGENT_ENTERPRISE_SANDBOX_GATEWAY_AUTH_TOKEN	empty	Optional X-Inner-Api-Key sent to the Enterprise Gateway.
DIFY_AGENT_ENTERPRISE_SANDBOX_GATEWAY_TIMEOUT	30	Enterprise control-plane timeout in seconds.
DIFY_AGENT_ENTERPRISE_SANDBOX_PROXY_TIMEOUT	60	Enterprise shellctl-proxy timeout in seconds.
DIFY_AGENT_E2B_API_KEY	empty	E2B API key; required for E2B.
DIFY_AGENT_E2B_TEMPLATE	difys-default-team/dify-agent-local-sandbox	Prepared E2B template containing shellctl and the deployment-default Home environment.
DIFY_AGENT_E2B_ACTIVE_TIMEOUT_SECONDS	3600	Maximum continuous active time for the RuntimeLease spanning one complete Agent run. Its default intentionally matches DIFY_AGENT_RUN_TIMEOUT_SECONDS, but the settings are independently configurable. Binding resources pause on timeout; this setting does not own the run terminal state and is not a retention TTL.
DIFY_AGENT_E2B_SHELLCTL_PORT	5004	shellctl port exposed by the E2B template.
DIFY_AGENT_SHELL_REDACT_PATTERNS	empty	JSON array of additional regex patterns redacted from Shell output.
DIFY_AGENT_STUB_API_BASE_URL	empty	HTTP(S) Agent Stub API base URL reachable from the Sandbox. It may be the service root or /agent-stub. Enables DIFY_AGENT_STUB_* env injection for user shell.run jobs.
DIFY_AGENT_SANDBOX_FILES_BASE_URL	empty	Dify API base URL reachable from the Sandbox for signed /files/* upload/download bytes, including Config file and skill pulls. Required when Agent Stub file operations are enabled. May include an ingress path prefix, but not a query or fragment.
DIFY_AGENT_STUB_UPLOAD_FILE_SIZE_LIMIT	50	Agent service-owned maximum Agent Stub upload size in MiB. The file-request handler factory converts it to bytes and sends it to Dify API as the required max_size used to sign a size-limited upload URL.
DIFY_AGENT_SERVER_SECRET_KEY	empty	Security-sensitive server-wide root secret used to derive the JWE encryption key for Agent Stub bearer tokens; required when DIFY_AGENT_STUB_API_BASE_URL is set. The supplied default config uses a development value; set a unique unpadded base64url 32-byte secret in production.
DIFY_AGENT_OUTBOUND_HTTP_CONNECT_TIMEOUT	10	Shared outbound HTTP connect timeout in seconds.
DIFY_AGENT_OUTBOUND_HTTP_READ_TIMEOUT	600	Shared outbound HTTP read timeout in seconds.
DIFY_AGENT_OUTBOUND_HTTP_WRITE_TIMEOUT	30	Shared outbound HTTP write timeout in seconds.
DIFY_AGENT_OUTBOUND_HTTP_POOL_TIMEOUT	10	Shared outbound connection-pool wait timeout in seconds.
DIFY_AGENT_OUTBOUND_HTTP_MAX_CONNECTIONS	100	Maximum total shared outbound HTTP connections.
DIFY_AGENT_OUTBOUND_HTTP_MAX_KEEPALIVE_CONNECTIONS	20	Maximum idle shared outbound HTTP connections.
DIFY_AGENT_OUTBOUND_HTTP_KEEPALIVE_EXPIRY	30	Idle keep-alive expiry in seconds.
DIFY_AGENT_TRAJECTORY_ENABLED	false	Opt-in switch for the separate Agent trajectory export pipeline.
DIFY_AGENT_APPLICATION_TRACING_ENABLED	false	Collect API-authorized application spans into the Dify inner API tracing inbox. Requires the server inner API URL and key.
DIFY_AGENT_TRAJECTORY_OTLP_TRACES_ENDPOINT	empty	Full HTTP(S) OTLP traces endpoint required when trajectory export is enabled.
DIFY_AGENT_TRAJECTORY_OTLP_HEADERS	{}	JSON object of request headers sent with Agent OTLP trace exports.
DIFY_AGENT_TRAJECTORY_SERVICE_NAME	dify-agent-trajectory	Resource service name attached to Agent trajectory spans.
DIFY_AGENT_TRAJECTORY_INCLUDE_CONTENT	false	Include message and tool content in Agent trajectory spans.
DIFY_AGENT_TRAJECTORY_TRACE_CONTEXT_MODE	isolated	isolated starts independent platform/Agent traces; shared preserves cross-instance parent context. Requires process restart to change.
DIFY_AGENT_TRAJECTORY_MAX_QUEUE_SIZE	2048	Maximum number of spans queued for Agent trace export; must be positive.
DIFY_AGENT_TRAJECTORY_MAX_EXPORT_BATCH_SIZE	512	Maximum spans per export batch; must be positive and no greater than the queue size.
DIFY_AGENT_TRAJECTORY_SCHEDULE_DELAY_MS	5000	Delay between scheduled Agent span exports, in milliseconds; must be positive.
DIFY_AGENT_TRAJECTORY_EXPORT_TIMEOUT_MS	5000	Batch processor export timeout setting, in milliseconds; must be positive. Does not change the OTLP HTTP exporter request timeout.

Platform observability uses the process-level Logfire instance and standard OTEL_* / LOGFIRE_* configuration. Agent observability uses a separate local Logfire instance and is disabled by default. Set DIFY_AGENT_TRAJECTORY_ENABLED=true and a full DIFY_AGENT_TRAJECTORY_OTLP_TRACES_ENDPOINT to opt in; DIFY_AGENT_TRAJECTORY_OTLP_HEADERS is a JSON object of request headers. Agent traces do not inherit platform endpoints, credentials, sampling, or console export. Message/tool content requires the separate DIFY_AGENT_TRAJECTORY_INCLUDE_CONTENT opt-in. Platform FastAPI parameter/error payloads, HTTPX bodies/headers and Redis statements are not captured by these instrumentations. The Agent instance is created at service startup and shut down after active runs finish. These are deployment-level settings, not yet per-tenant authorization or routing; a shared backend does not provide storage isolation.

With DIFY_AGENT_TRAJECTORY_TRACE_CONTEXT_MODE=isolated (the default), Agent run spans start independent traces instead of inheriting platform request parents. Platform HTTPX/Redis instrumentation also detaches local Agent parents while preserving platform-only and incoming distributed trace relationships. In shared mode both pipelines retain cross-instance parent context and the Agent sampler respects the parent sampling decision. Use shared only when both pipelines export to the same queryable backend space; otherwise missing-root spans can recur. Service names do not select the mode. The two SDK instances, exporters, enable switch, and content opt-in remain separate. The mode is fixed at process initialization and requires a restart to change. No extra shared root span is added; incomplete traces may still appear temporarily while a run or export is in progress.

Application tracing is a third, independent pipeline. When enabled, a run's API-issued execution_context.tracing supplies its durable run/invocation IDs, OpenTelemetry parent, session and content/sampling policy. Its spans are sent as JSON to POST /inner/api/agent/tracing/batches, authenticated with the server's DIFY_AGENT_INNER_API_KEY in X-Inner-Api-Key. Provider credentials and routing remain in the Dify API. Application spans do not inherit platform or deployment trajectory destinations, and one Agent is never instrumented by both application and deployment pipelines. Binary content is excluded.

The actual model-call span also carries the Dify gateway's billed amount as dify.llm.total_price (a decimal string) and dify.llm.currency when a usage payload is returned. gen_ai.response.time_to_first_token is measured in seconds (dify.llm.time_to_first_token_unit=s): gateway-provided TTFT takes precedence; otherwise it measures request issue to the first content/tool-call chunk. dify.llm.time_to_first_token_source distinguishes those two sources. Missing accounting is left absent. The current Graphon LLM gateway does not expose cache token counters, so the Agent adapter does not manufacture them.

Each batch has a stable batch_id. Retries reuse the same payload, and a 2xx response is accepted only with {result: "success", batch_id, accepted_spans} matching that batch. An invocation emits an empty final: true batch after its ended spans are flushed and acknowledged, before its normal terminal event when delivery completes within five seconds. Failure or cancellation still flushes. Collection failures do not fail the Agent run; outstanding delivery can finish in the background after the terminal-event deadline. A full in-memory queue or unacknowledged data suppresses completion instead of claiming a complete trace. The queue is bounded (2048 spans, batches of at most 128), with up to 128 unacknowledged batches retained until invocation finalization. Batches are split below 1 MiB of UTF-8 JSON. A single oversized span is dropped and marks its invocation incomplete rather than silently truncating content or sending a success marker. This is not a durable local spool: process loss can leave an incomplete trace in the API. Shutdown drains the collector after active runs finish.

If setup fails or cancellation arrives before Pydantic AI starts its Agent span, the collector emits one failed invoke_agent span with the measured setup interval, then finalizes normally. It records dify.agent.execution_state as initialization_failed or cancelled. Runs that already started their native Agent span keep that span without an extra wrapper. Sampling-off invocations remain span-free; successful but unexplained empty collections are not converted to synthetic success.

Example .env:

DIFY_AGENT_REDIS_URL=redis://localhost:6379/0
DIFY_AGENT_REDIS_PREFIX=dify-agent-dev
DIFY_AGENT_SHUTDOWN_GRACE_SECONDS=30
DIFY_AGENT_RUN_RETENTION_SECONDS=7200
DIFY_AGENT_RUN_EVENT_STREAM_MAX_LENGTH=5000
DIFY_AGENT_STREAM_TEXT_DELTA_FLUSH_INTERVAL_MS=100
DIFY_AGENT_STREAM_TEXT_DELTA_MAX_CHARS=4096
DIFY_AGENT_RUN_TIMEOUT_SECONDS=3600
DIFY_AGENT_API_TOKEN=replace-with-agent-backend-token
DIFY_AGENT_PLUGIN_DAEMON_URL=http://localhost:5002
DIFY_AGENT_PLUGIN_DAEMON_API_KEY=replace-with-daemon-key
DIFY_AGENT_INNER_API_URL=http://localhost:5001
DIFY_AGENT_INNER_API_KEY=replace-with-dify-inner-api-key-for-plugin
DIFY_AGENT_RUNTIME_BACKEND=local
DIFY_AGENT_LOCAL_SANDBOX_ENDPOINT=http://127.0.0.1:5004
DIFY_AGENT_LOCAL_SANDBOX_AUTH_TOKEN=replace-with-shellctl-token
# Set these when shellctl runs directly on a host that does not have /home/dify.
DIFY_AGENT_LOCAL_SANDBOX_MATERIALIZED_HOME_ROOT=/tmp/dify-agent/materialized-homes
DIFY_AGENT_LOCAL_SANDBOX_WORKSPACE_ROOT=/tmp/dify-agent/workspaces
DIFY_AGENT_LOCAL_SANDBOX_HOME_SNAPSHOT_ROOT=/tmp/dify-agent/home-snapshots
DIFY_AGENT_STUB_API_BASE_URL=https://agent.example.com/agent-stub
DIFY_AGENT_SANDBOX_FILES_BASE_URL=https://dify.example.com
DIFY_AGENT_STUB_UPLOAD_FILE_SIZE_LIMIT=50
# This is security-sensitive: it derives the JWE encryption key for Agent Stub bearer tokens.
# Replace this development default in production.
# Generate one with: python -c 'import secrets; print(secrets.token_urlsafe(32))'
DIFY_AGENT_SERVER_SECRET_KEY=MDEyMzQ1Njc4OWFiY2RlZjAxMjM0NTY3ODlhYmNkZWY

The two Sandbox-facing base URLs have different owners. Agent Stub control requests use DIFY_AGENT_STUB_API_BASE_URL; signed file bytes use DIFY_AGENT_SANDBOX_FILES_BASE_URL. DIFY_AGENT_INNER_API_URL remains a trusted service-to-service URL and is never returned to the Sandbox.

Config file and skill pulls use the same split: Agent Stub authorizes the Config target and returns a short-lived URL, then the Sandbox fetches the bytes directly from the Dify API /files/* data plane.

Removing Agent Stub gRPC is a breaking transport migration: replace every grpc:// Agent Stub URL with HTTP(S), remove DIFY_AGENT_STUB_GRPC_BIND_ADDRESS, and deploy without a gRPC fallback.

For a remote Sandbox, expose only /agent-stub/* from Agent Backend and the existing /files/* Dify API data plane. The /files/* ingress must preserve the complete signed query string and set its request-body limit above the configured file-size limit to leave room for multipart framing and headers; the two limits need not be numerically equal. Use response streaming and timeouts suitable for large downloads. Do not expose Agent Backend /runs, Workspace, or Binding management routes through the Sandbox ingress.

Browser presentation URLs are independent. Configure Dify API FILES_URL to a browser-reachable public origin, or leave it empty so responses use same-origin relative /files/... URIs. Never set FILES_URL to a Docker-only service name such as http://api:5001.

DIFY_AGENT_SHELLCTL_ENTRYPOINT and DIFY_AGENT_SHELLCTL_AUTH_TOKEN remain accepted only as legacy aliases for the two Local settings. New deployments must use DIFY_AGENT_LOCAL_SANDBOX_ENDPOINT and DIFY_AGENT_LOCAL_SANDBOX_AUTH_TOKEN. There is no compatibility setting for the removed shell-provider selector.

Local shellctl deployments must start with shellctl serve --require-auth-token and set SHELLCTL_AUTH_TOKEN to the same secret as the Agent Backend. Compose's local_sandbox command enables this check; missing, empty, or whitespace-only tokens fail before service initialization. E2B and Enterprise retain their existing token requirements, and an unset Local endpoint still disables runtime.

For Compose, set DIFY_AGENT_LOCAL_SANDBOX_AUTH_TOKEN in a shared interpolation source: the invoking shell, docker/.env, or a Compose --env-file. This supplies the same secret to Agent Backend and shellctl. Setting it only in a service env file is insufficient: Compose's explicit environment values override that file, including with an empty interpolated value. The legacy token alias remains accepted. Deploy this change with a rebuilt runtime image containing --require-auth-token; the previously published 1.17.1 image does not support the new flag.

When shellctl path isolation is enabled, any Landlock setup error aborts the job with exit code 125 before its user script executes. Deployments may explicitly set SHELLCTL_ENABLE_PATH_ISOLATION=false to disable shellctl path isolation.

The backend selection is deployment-private. Shell-enabled run requests use an Execution Context, dify.runtime, and dify.shell graph. Runtime config carries only the opaque backend_binding_ref resolved by Dify API. See Runtime resources for the ownership and lifecycle contract.

Run records and event streams use the same retention. Status writes refresh the record TTL, and event writes refresh both the stream TTL and the corresponding record TTL so active runs that keep producing events remain observable. Text deltas for the same response part and provider metadata are coalesced within a soft debounce/size window before being published. The debounce timer flushes when reading the next source event would block; already-ready events may keep merging until the character threshold or a non-compatible event is reached. Every Redis Stream write also applies the configured approximate maximum length. A reconnect cursor older than the retained window resumes from the oldest remaining event, so callers must treat the terminal snapshot and application-owned history as authoritative rather than relying on the Agent event stream as long-term history.

Validate the E2B Compose deployment

The E2B overlay requires a prepared template that starts shellctl on port 5004. The default is difys-default-team/dify-agent-local-sandbox. It also requires an E2B API key in DIFY_AGENT_E2B_API_KEY; the Compose interpolation accepts E2B_API_KEY and E2B_API_TOKEN only as deployment-level fallbacks.

From the repository root, keep the normal docker/.env unchanged and export the secret for this shell:

export DIFY_AGENT_E2B_API_KEY="${DIFY_AGENT_E2B_API_KEY:-${E2B_API_KEY:-${E2B_API_TOKEN:-}}}"
test -n "$DIFY_AGENT_E2B_API_KEY"
export DIFY_AGENT_E2B_TEMPLATE=difys-default-team/dify-agent-local-sandbox

docker compose \
  --env-file docker/.env \
  -f docker/docker-compose.yaml \
  -f docker/docker-compose.e2b.yaml \
  up -d --build

The overlay builds dify-api:e2b-local and dify-agent-backend:e2b-local from the current checkout. It disables the normal Local sandbox service, switches Dify Agent to E2B, and mounts PostgreSQL on the separate dify_e2b_postgres_data Compose volume. That database is empty when the volume is first created and is isolated from the normal stack; later starts reuse it until an operator explicitly removes the volume.

Verify the merged deployment and the branch-built Dify Agent API:

docker compose \
  --env-file docker/.env \
  -f docker/docker-compose.yaml \
  -f docker/docker-compose.e2b.yaml \
  ps

agent_backend_port="$(
  docker compose \
    --env-file docker/.env \
    -f docker/docker-compose.yaml \
    -f docker/docker-compose.e2b.yaml \
    port agent_backend 5050 | awk -F: 'NR == 1 { print $NF }'
)"
test -n "$agent_backend_port"
curl --fail --silent --show-error \
  --connect-timeout 2 --max-time 5 \
  --retry 12 --retry-delay 1 --retry-connrefused --retry-max-time 60 \
  "http://127.0.0.1:${agent_backend_port}/openapi.json" \
  >/dev/null

docker compose \
  --env-file docker/.env \
  -f docker/docker-compose.yaml \
  -f docker/docker-compose.e2b.yaml \
  logs --tail=100 agent_backend api worker

Stop the validation stack without deleting its isolated database volume:

docker compose \
  --env-file docker/.env \
  -f docker/docker-compose.yaml \
  -f docker/docker-compose.e2b.yaml \
  down

DIFY_AGENT_E2B_ACTIVE_TIMEOUT_SECONDS controls continuous active E2B time. The physical resource behind a Binding pauses when that timeout fires, preserving the current Workspace. The setting is not a resource-age TTL, does not delete paused resources or immutable snapshots, and does not authoritatively finalize the Agent run. If the paused Sandbox is first observed by a Shell tool, that provider failure is returned to Pydantic AI as a tool error observation.

The run and E2B defaults both equal 3600 seconds, but independent clocks and asynchronous E2B pause propagation make their ordering nondeterministic. A Shell provider RuntimeError observed first becomes a tool observation. In contrast, run-deadline cancellation propagates through the Shell boundary; only the Dify Agent run deadline owns the terminal agent_run_limit_exceeded failure.

The E2B backend uses SDK 2.55.1 or later with three control-plane retries. The SDK retries 502 only for replayable operations, including when the response has Retry-After; a gateway can return 502 after a resource was already created. Sandbox and snapshot creation therefore surface 502 without replaying the request. The SDK still retries 503, 429 with Retry-After, and safely replayable network failures. Backend adapters invoke each SDK operation once; exhausted retries and timeouts do not start a second SDK request budget. Resource creation and snapshot creation are also not replayed after an ambiguous post-write network failure. See the SDK replay policy.

This policy covers E2B lifecycle API calls. Shell commands and normal file operations use the separate shellctl client, and usage collection uses its own HTTP client. Neither inherits the E2B SDK retry or connection-pool settings.

The backend explicitly requests mode="full" for timeout pauses, explicit pauses, and Home Snapshots, preserving both filesystem and memory state. Binding acquisition uses on_resume="restore"; connect does not accept a snapshot mode. These settings preserve the existing full-memory restore intent while replacing the deprecated keep_memory option. They do not override platform fallbacks: E2B may save only the filesystem if a timeout pause is continuously refused because a previous snapshot is still being finalized. See the E2B persistence contract.

OpenShell backend

See the OpenShell Runtime Backend guide for its configuration reference, runtime image build instructions, gateway and shared-volume setup, and deployment validation.

Run runtime-backend integration contracts

Run the disposable Local contract from the dify-agent directory. The script starts one local-sandbox container on an unused port and removes that exact container on exit. It intentionally has no image fallback: provide a Local Sandbox image built from the same commit as the code under test so an older shellctl implementation cannot make the lifecycle contract pass incorrectly:

cd dify-agent
DIFY_AGENT_TEST_LOCAL_SANDBOX_IMAGE=<same-commit-local-sandbox-image> \
  tests/integration/dify_agent/runtime_backend/run_local_integration.sh

To use an already managed Local shellctl endpoint instead:

cd dify-agent
DIFY_AGENT_TEST_LOCAL_SHELLCTL_ENDPOINT=http://127.0.0.1:5004 \
DIFY_AGENT_TEST_LOCAL_SHELLCTL_AUTH_TOKEN=replace-with-shellctl-token \
  pdm run pytest --import-mode=importlib \
  tests/integration/dify_agent/runtime_backend/test_working_environment.py \
  -k local -q -rs

Run the real E2B contract with an explicit test credential and template:

cd dify-agent
DIFY_AGENT_TEST_E2B_API_KEY="$E2B_API_TOKEN" \
DIFY_AGENT_TEST_E2B_TEMPLATE=difys-default-team/dify-agent-local-sandbox \
  pdm run pytest --import-mode=importlib \
  tests/integration/dify_agent/runtime_backend/test_working_environment.py \
  -k e2b -q -rs

For OpenShell, see Run the OpenShell integration contract.

The Local auth token is optional when shellctl has authentication disabled. The E2B contract uses the one-hour E2B_MAX_ACTIVE_TIMEOUT_SECONDS RuntimeLease limit. This is continuous active test time, not a post-test retention TTL. All contracts create unique resources and perform explicit cleanup in finally blocks.

Scheduling and shutdown semantics

POST /runs persists a running run record and starts an asyncio task in the same process. There is no Redis job stream, consumer group, pending reclaim, or automatic retry layer. Request-shaped runtime failures such as bad composition, prompt, output, or snapshot inputs are reported later as failed runs rather than rejected synchronously once the request DTO itself is accepted.

Each run explicitly limits Pydantic AI to 500 model-request steps. Tool calls do not have a separate count limit, but every model request used to continue the tool loop consumes one of those steps.

DIFY_AGENT_RUN_TIMEOUT_SECONDS additionally applies a wall-clock deadline around native Pydantic AI's agent.run(...). This includes capability entry, RuntimeLease acquisition, dynamic tool preparation, the model/tool loop, and the event handler. Request validation and agent assembly happen before this deadline. Expiry cancels the native run; capability scopes unwind and release resources before the runner generates the session snapshot and finalizes the run as failed with error_type: "agent_run_limit_exceeded". History persistence, snapshot generation, and terminal finalization occur after native execution.

During FastAPI shutdown the scheduler rejects new runs, waits up to DIFY_AGENT_SHUTDOWN_GRACE_SECONDS for active tasks, then cancels remaining tasks and attempts to finalize them as failed. Success and failure use an atomic Redis transition. Cancellation first atomically records a private intent; after the owner exits the runner, a second atomic transition appends run_cancelled, updates the run record, and deletes the intent. The first accepted success, failure, or cancellation intent wins. A hard process crash can still leave active runs, including runs with accepted cancellation intent, stuck as running; there is no in-service recovery or worker handoff.

Horizontal scaling is possible by running multiple API processes against the same Redis prefix, but each process executes only the runs it accepted. Redis provides shared status/event visibility, not load balancing or queued-job recovery. The cancel endpoint can atomically accept a running run on any process. The process that owns the runner observes the private cancellation-intent stream, cancels and cleans up its local task, and only then emits run_cancelled. The HTTP response confirms that cancellation intent is durable; GET /runs/{run_id} may still report running until cleanup finishes. Retrying an accepted or completed cancellation is idempotent.

Atomic terminal finalization currently assumes the configured Redis URL targets one Redis deployment that can execute all run-coordination keys in a Lua script. The record and event key names are unchanged, and cancellation adds a private cancel-intent key. These keys do not contain a shared Redis Cluster hash tag, so Redis Cluster is not supported for this transition. During a rolling upgrade, older processes can still use the former split event/status writes; treat the single-terminal invariant as active only after those processes have exited. Deploy atomic terminal finalization everywhere first, then ensure every process that can own a runner has the cancellation observer before relying on route-independent cancellation. Operators should then alert on more than one terminal event per run and on disagreement between the run record status and terminal event type.

Run inputs and session snapshots

Submit a RunComposition with registered module names and JSON Config. The llm model module is required and references an execution_context module. Prompt modules contribute instructions and config.user input:

{
  "composition": {
    "schema_version": 2,
    "layers": [
      {"name": "prompt", "config": {"prefix": "You are concise.", "user": "Summarize the current state."}},
      {"name": "execution_context", "config": {"tenant_id": "tenant-id", "agent_mode": "agent_app", "invoke_from": "web-app"}},
      {"name": "llm", "config": {"plugin_id": "langgenius/openai", "model_provider": "openai", "model": "gpt-4o-mini"}}
    ]
  }
}

config.user can be a string or a list of strings. Empty or whitespace-only effective prompts and module configuration are validated by the runner before resource acquisition or model I/O. HTTP create-run parses the wire shape, persists the run and schedules execution; configuration errors produce run_failed.

Agent App callers can use the agent_app_user_prompt module to send text and images in the same model turn. Each image must provide exactly one transport: an HTTP(S) url or unprefixed Base64 data in base64_data. The image is passed as structured multimodal content; it is not interpolated into config.text.

{
  "name": "agent_app_user_prompt",
  "config": {
    "text": "Describe this image.",
    "files": [
      {
        "delivery": "multimodal",
        "type": "image",
        "filename": "earth.png",
        "mime_type": "image/png",
        "format": "png",
        "url": "https://files.example.com/earth.png",
        "base64_data": null,
        "detail": "high"
      }
    ]
  }
}

All attachments use config.files. The delivery field selects how an attachment is presented, independently of its file type (image, document, audio, video, or custom):

• delivery: "multimodal" currently supports type: "image" only. Dify API chooses URL or Base64 transport according to MULTIMODAL_SEND_FORMAT.
• delivery: "download" preserves the original file type, including image when the selected model has no Vision feature. It contains either transfer_method: "remote_url" with url, or a local_file, tool_file, or datasource_file transfer method with a canonical reference. An optional filename identifies the attachment in the rendered file locators.

When an Agent App has KnowledgeFS spaces, each uploaded local_file image sent as multimodal content also has a named download entry. The model receives the image directly and can use its canonical reference with dify-agent knowledge search --image-file-id. Remote URLs and tool/datasource file references are not valid upload references for KnowledgeFS image search.

The module appends sandbox file-download instructions for download attachments and adds multimodal attachments as structured model content. Callers keep config.text as the original user text. Omitting config.files is equivalent to an empty list.

The optional history module uses the reserved name history. It persists captured messages, including failure and cancellation history, with current instructions removed. Snapshots have schema_version: 2 and a layers JSON dict containing only State. Submit current Config on each turn along with the previous snapshot. Modules can be added or removed between turns; retained state is matched by name.

Known v1 snapshots are converted once at read time without restoring legacy instances or lifecycle. Native resource scopes always release leases and clear tracked shell jobs at run exit. Success always contains the resulting snapshot; failure and cancellation contain one when validated run data was available and cleanup produced a snapshot. Callers retain the previous snapshot when none is returned. Product Workspace retirement remains owned by the API.

Observing runs

Use the HTTP status endpoint for coarse state and the event endpoints for detailed progress:

• POST /runs creates a running run and schedules it locally.
• GET /runs/{run_id} returns running, succeeded, failed, or cancelled. Failed records can also expose a stable machine-readable error_type alongside the diagnostic error text.
• POST /runs/{run_id}/cancel atomically accepts cancellation on any API process and returns immediately. CancelRunResponse.status == "cancelled" acknowledges a durably accepted cancellation intent, not completed runner cleanup. Callers that require cleanup-complete state or its session snapshot must await the public run_cancelled event or use cancel_run_and_wait. The endpoint returns 409 only when a success/failure terminal already won.
• GET /runs/{run_id}/events polls the Redis Stream event log with after and next_cursor cursors.
• GET /runs/{run_id}/events/sse replays and streams events over SSE. The SSE id is the event Redis Stream ID. after query cursors take precedence over Last-Event-ID headers. The server closes the SSE response normally after delivering a terminal event. Clients must stop reconnecting after consuming that event. Both cursor forms remain exclusive resume cursors, so the server does not resend a terminal event that the supplied cursor already excludes.

Both event endpoints replay with nonblocking XREAD, which supports exclusive resume cursors on Redis 6.0, including 6.0.14. 0-0 starts at the earliest retained event. Polling returns at most limit events; an empty page preserves the supplied after as next_cursor. SSE switches to blocking reads after replay is exhausted. Both endpoints return 404 when the run record is missing. This read-command compatibility does not define the platform's full Redis support matrix; the Redis deployment constraints above still apply.

Successful runs emit run_started, zero or more pydantic_ai_event, and run_succeeded. Failed runs end with run_failed, and accepted cancellations end with run_cancelled. Each run can append at most one of these terminal events. Event envelopes retain id, run_id, type, data, and created_at; data is typed per event type, including Pydantic AI's AgentStreamEvent payload for pydantic_ai_event and a terminal event may contain a SessionSnapshot for resumption. run_succeeded always contains it; run_failed and run_cancelled contain it when the runner produced State after native capability cleanup. A successful run contains JSON-safe output, which may be null. Failed event payloads contain the diagnostic error, optional source-specific reason, optional stable error_type, and optional session_snapshot. Cancelled payloads likewise may contain session_snapshot. Pydantic AI request/step budget exhaustion enforced by Dify Agent is reported as error_type: "agent_run_limit_exceeded"; consumers should branch on that value rather than parsing the error text. The Dify Agent-owned wall-clock run deadline uses the same error type; provider and connection timeouts do not. The matching failed run record and terminal event are committed atomically with the same error type. For independently deployed Agent backend and API services, deploy consumers that accept the optional field before producers begin emitting it because the public protocol models reject unknown fields.

Examples

The repository includes simple consumers that print observed output/events:

• dify-agent/examples/dify_agent/dify_agent_examples/run_server_consumer.py creates a run and polls events.
• dify-agent/examples/dify_agent/dify_agent_examples/run_server_sse_consumer.py consumes raw SSE frames for an existing run id.

The create-run examples submit Dify plugin model modules, so they require Redis, the Agent server, Dify API gateway settings, and a configured model provider in Dify.

Automatic KnowledgeFS quality reporting

Knowledge CLI calls now contribute bounded, server-owned investigation observations. The Stub records command IDs, queries/paths, outcomes, error codes, timing, small text snippets and receipt IDs. The runtime separately marks which observations were delivered to the model. This ledger survives the consumable tool-result queue and session cleanup. It shares the knowledge budget's two-hour retention and identity boundary. A human-input pause retains the investigation; a resumed run continues it. A new user turn gets a fresh investigation. Images and bearer capabilities are never included in these observations.

After the authoritative final-answer event, the runtime submits the original question, final answer and observations to the private Dify API. Failed runs submit an interrupted report; cancellation leaves a bounded diagnostic ledger and does not create quality cases. A human-input pause does not submit a final report. Queue admission is retried with the same investigation ID, and submission failures do not change the completed Agent result.

The existing dataset Celery workers process one job per investigated knowledge space. Each attempt reconstructs the Agent/app/configuration ownership and reissues a capability. An authorization fingerprint binds observations to the original caller, ACL/content policy revisions and content scopes; a changed scope rejects publication of old observations. Agent applications, Workflow Agent nodes and interactive previews use their actual caller identities and a dedicated queries.agent_investigation.capture action.

KnowledgeFS makes one batched assessment of the space's complete investigation using its configured reasoning model with a 60-second model budget. The worker uses the existing KNOWLEDGE_FS_RETRIEVAL_TEST_TIMEOUT_SECONDS transport timeout for this operation. Query rewrites and recovered empty searches are grouped. An unresolved subquestion produces a failed-query record; only retrieval-miss produces a quality bad case tagged agent, auto-captured, retrieval-miss. Coverage gaps, unrelated questions and uncertain issues do not become retrieval bad cases. Tool errors, directory probing, unobserved results, image-only investigations and interruptions cannot by themselves produce a bad case. If the model is unavailable or returns an invalid assessment, the trace is retained as uncertain without automatic cases.

The committed assessment and stable child IDs fence retries, so projection retries do not create duplicate failed queries/cases or repeat the committed judgment. Separate unanswered subquestions remain separate, with at most eight issues per space. The original question and bounded command history remain linked to each issue trace. The final answer is used for assessment but is not copied into a knowledge space's shared trace, since it may contain content from other spaces. Document snippets likewise remain transient classifier input; shared traces retain command metadata and receipt IDs without copying document text outside the existing evidence permission projection. Reports are best-effort: after bounded dispatch retries, an undelivered report remains only in the expiring runtime ledger. Deploy matching Agent, Dify API/worker and KnowledgeFS versions to enable the complete path; no new environment variable or Agent tool is required.