MCP Quality in CI

NAT MCP Quality v0.1 is designed to run as a deterministic CI quality gate without an LLM.

Exit codes

The frozen MCP CLI contract is:

Exit code	Meaning
`0`	Command completed and the quality gate passed.
`1`	Command completed and the target quality gate failed.
`2`	Runtime/dependency failure prevented a valid determination.
`3`	Existing NAT configuration-error compatibility where applicable.

A target-server quality failure is therefore different from a broken CI environment or an unavailable dependency.

GitHub Actions example

name: MCP Quality
 
on:
  pull_request:
  push:
    branches: [main]
 
jobs:
  mcp-quality:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
 
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
 
      - name: Install NAT with MCP support
        run: python -m pip install "nat-engine[mcp]"
 
      - name: Inspect MCP contract
        run: |
          nat mcp inspect \
            --url "${{ secrets.MCP_TEST_URL }}" \
            --output json > mcp-inspect.json
 
      - name: Functional MCP gate
        run: |
          nat mcp test \
            --url "${{ secrets.MCP_TEST_URL }}" \
            --output json > mcp-functional.json
 
      - name: Security MCP gate
        run: |
          nat mcp security \
            --url "${{ secrets.MCP_TEST_URL }}" \
            --output json > mcp-security.json
 
      - name: Upload MCP artifacts
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: mcp-quality
          path: mcp-*.json

If you are validating an unreleased source checkout, replace the install command with:

python -m pip install -e ".[mcp]"

Conformance as a separate gate

Official MCP conformance can be run independently because its dependency/execution semantics are different from NAT's deterministic functional checks:

- name: Official MCP conformance
  run: |
    nat mcp conformance \
      --url "${{ secrets.MCP_TEST_URL }}" \
      --output json > mcp-conformance.json

An upstream runner/dependency problem returns an invalid execution result rather than a false target success.

Preserve JSON artifacts

For the first design-partner cohort, JSON files are the intended persistence boundary. Archive them with the build or test run rather than requiring an MCP-specific database.

Useful artifacts include:

mcp-inspect.json
mcp-functional.json
mcp-conformance.json
mcp-security.json

The inspection artifact includes the normalized snapshot and fingerprint. Test/conformance/security artifacts contain structured NAT-native results and gate summaries appropriate for downstream reporting.

Recommended design-partner workflow

Use an MCP endpoint dedicated to validation, run inspection first, then functional/security checks, and add conformance when the upstream runner is available in the CI environment. Retain the JSON artifacts with the commit/build that produced them.

Do not convert exit code 2 into a target quality failure. It means NAT could not make a valid determination and the CI environment or dependency should be investigated separately.

Was this helpful?

Overview Security & Redaction