MCP Quality v0.1

NAT MCP Quality v0.1 provides deterministic quality engineering for Model Context Protocol (MCP) servers. It discovers an MCP server contract, normalizes and fingerprints it, runs deterministic functional and security checks, invokes the official MCP conformance runner through an adapter, and emits human-readable or JSON results suitable for CI.

MCP Quality v0.1 does not require an LLM or paid model API.

Install

For a source checkout during design-partner validation:

python -m pip install -e ".[mcp]"

For a published release:

python -m pip install "nat-engine[mcp]"

Verify the MCP command group:

nat mcp --help

Inspect an MCP server

nat mcp inspect --url http://localhost:3000/mcp

Inspection reports the discovered server identity, protocol version, transport, tool/resource/prompt counts, and normalized SHA-256 contract fingerprint.

Use JSON output when you want to archive or process the result:

nat mcp inspect \
  --url http://localhost:3000/mcp \
  --output json > mcp-inspect.json

The fingerprint excludes volatile capture metadata and is designed to remain stable when semantically equivalent discovered collections arrive in a different order.

Run deterministic functional checks

nat mcp test --url http://localhost:3000/mcp

The default functional run is intentionally non-destructive. NAT validates discovered contracts but does not invent tool arguments or invoke arbitrary tools automatically. Explicit tool invocation and resource-read operations exist in the shared MCP execution/service layer for configured tests.

Machine-readable output:

nat mcp test \
  --url http://localhost:3000/mcp \
  --output json > mcp-functional.json

Run official MCP conformance

nat mcp conformance --url http://localhost:3000/mcp

NAT delegates protocol conformance to the official upstream MCP conformance runner and normalizes its structured results into NAT-native result objects. NAT does not maintain a private fork of the authoritative conformance suite.

If the runner is unavailable or cannot produce a valid determination, NAT reports an execution/dependency failure instead of treating the server as conformant.

nat mcp conformance \
  --url http://localhost:3000/mcp \
  --output json > mcp-conformance.json

Run the deterministic security baseline

nat mcp security --url http://localhost:3000/mcp

The bounded v0.1 security catalog includes deterministic checks for:

• duplicate or ambiguous tool names after normalization;
• obvious credential-like material in tool descriptions;
• invalid resource URIs;
• default discovery of local file:// paths commonly associated with secrets or credentials;
• breaking contract drift when a baseline is supplied through the shared service layer.

nat mcp security \
  --url http://localhost:3000/mcp \
  --output json > mcp-security.json

See Security & Redaction for the v0.1 trust boundary and non-goals.

Result categories

MCP Quality keeps three result classes distinct:

• Protocol / conformance — official MCP protocol correctness results.
• Functional — deterministic contract and explicitly configured operation checks.
• Security — deterministic trust/security checks that do not require model reasoning.

General autonomous-agent behavioral evaluation begins after v0.1 and is not part of this release.

CI and JSON artifacts

All public MCP commands support --output json. The command exit status distinguishes a target quality failure from a run that could not make a valid determination. See CI Integration for examples and the frozen exit-code contract.

v0.1 boundaries

The first design-partner validation intentionally uses CLI + CI + JSON artifacts. MCP-specific FastAPI routes and database persistence are deferred unless validation demonstrates that centralized execution or persisted run retrieval is required.

v0.1 does not include AgentSpec, LLM-as-judge evaluation, semantic goal scoring, cross-model comparison, production trace ingestion, chaos testing, model-in-the-loop tool-poisoning analysis, new SaaS/billing packaging, or dashboard redesign.