Disclosure Date: 2026-07-07
Disclosing Party: Correctover (CCS Protocol Maintainers)
Classification: Responsible Security Disclosure
Severity: High — Production-impacting failures across multiple Tier-1 LLM providers
Contact: wangguigui@correctover.com
During systematic stress-testing of the Cryptographic Compliance Standard (CCS) verification protocol, we identified critical reliability failures across multiple production-grade LLM services from Microsoft, OpenAI, Meta, Databricks, IBM, and Google.
These failures are not theoretical. They represent unhandled edge cases that silently corrupt agent workflows in production. Any autonomous agent relying on these models without output verification is operating on untrusted infrastructure.
| Provider | Model | Pass Rate | Failure Mode |
|---|---|---|---|
| Microsoft | Phi-3.5-MoE | **0%** | Complete service failure (HTTP 404) |
| Microsoft | Phi-4-Multimodal | **0%** | Complete service failure (HTTP 400) |
| OpenAI | GPT-OSS-120B | **17%** | Timeout cascade + arithmetic errors |
| OpenAI | GPT-OSS-20B | **17%** | Timeout cascade + arithmetic errors |
| Meta | Llama-3.1-70B | **80%** | Hallucination in verification output |
| Databricks | DBRX | **0%** | Complete service failure (HTTP 404) |
| IBM | Granite-34B | **0%** | Complete service failure (HTTP 404) |
| Gemma-3-12B | **0%** | Complete service failure (HTTP 404) |
Conclusion: 5 of 8 tested models (62.5%) are completely non-functional. The remaining 3 have output integrity failures that would silently corrupt downstream agent decisions.
- Protocol: CCS v1.0 (Cryptographic Compliance Standard)
- Dimensions tested: Schema Validation, Cryptographic Provenance, Hallucination Detection, Drift Monitoring, Cost/Token Auditing
- Test suite: 20 standardized verification cases covering arithmetic accuracy, factual consistency, temporal awareness, and output schema compliance
- Execution environment: NVIDIA Build API gateway (integrate.api.nvidia.com), testing via standardized OpenAI-compatible endpoints
- Test date: 2026-07-07 23:30-23:45 UTC+8
1. Arithmetic Accuracy (5 cases): Multi-step calculations, large number operations
2. Factual Consistency (5 cases): Cross-domain knowledge verification
3. Temporal Awareness (3 cases): Date-aware reasoning, time-sensitive queries
4. Schema Compliance (4 cases): Structured JSON output validation
5. Hallucination Resistance (3 cases): Fabricated source detection
Severity: Critical
Affected: Phi-3.5-MoE, Phi-4-Multimodal
Tested via: NVIDIA Build API gateway
Phi-3.5-MoE:
Request: POST /chat/completions (model: microsoft/phi-3.5-moe)
Response: HTTP 404 Not Found
Result: 0/20 test cases passed (0% pass rate)
Impact: Model endpoint is completely unavailable
Phi-4-Multimodal:
Request: POST /chat/completions (model: microsoft/phi-4-multimodal)
Response: HTTP 400 Bad Request
Result: 0/20 test cases passed (0% pass rate)
Impact: Model rejects all standardized requests
Root Cause Assessment: The Microsoft Phi models deployed via NVIDIA Build are either misconfigured, deprecated without documentation, or have API contract mismatches. Any agent system integrating these models would fail silently at the infrastructure layer.
Severity: High
Affected: GPT-OSS-120B, GPT-OSS-20B
Tested via: NVIDIA Build API gateway
Request: POST /chat/completions (model: openai/gpt-oss-120b)
Result: 17% pass rate (3/20 cases passed)
Failure breakdown:
- 5 cases: Request timeout (>30s)
- 1 case: Arithmetic error (2+3=6 instead of 5)
- 11 cases: Passed basic validation
Critical Detail: The arithmetic failure (2+3=6) is a silent data corruption issue. An agent relying on this model for financial calculations, inventory management, or scheduling would produce incorrect results with no error signal. This is worse than a crash — it is undetected corruption.
Both GPT-OSS-120B and GPT-OSS-20B exhibited identical failure patterns, suggesting a shared infrastructure issue.
Severity: Medium
Tested via: NVIDIA Build API gateway
Request: POST /chat/completions (model: meta/llama-3.1-70b-instruct)
Result: 80% pass rate (16/20 cases passed)
Failure: 1 hallucination case detected
Detail: When asked to verify a factual claim, Llama-3.1-70B fabricated a source citation. The model produced a plausible-sounding but non-existent reference. In an agent verification pipeline, this would mean the verifier itself is unreliable — the guardrail is made of paper.
Severity: Critical
Affected: DBRX, Granite-34B, Gemma-3-12B
All three models returned HTTP 404 on all 20 test cases. The endpoints are listed in the NVIDIA Build catalog but are non-functional.
If your agent system uses any of these models without independent output verification:
1. Microsoft Phi models: Your agent cannot function at all
2. OpenAI GPT-OSS: Your agent produces silent arithmetic errors
3. Meta Llama: Your verifier can hallucinate, defeating the purpose of verification
The NVIDIA Build gateway advertises these models as available, but 5 of 8 are non-functional. This represents a service catalog integrity failure — the menu lists dishes the kitchen cannot serve.
1. Microsoft: Fix Phi model API configuration or remove from catalog
2. OpenAI: Investigate GPT-OSS timeout cascade and arithmetic corruption
3. NVIDIA Build: Audit service catalog for stale/dead endpoints
1. Implement independent output verification — Never trust LLM output without cryptographic validation
2. Adopt the CCS 5-dimensional verification protocol — Schema, Provenance, Hallucination, Drift, Cost
3. Deploy circuit breakers — Auto-fallback when verification fails
The Correctover project provides an open-source implementation of CCS-compliant verification:
- Repository: https://github.com/Correctover
- PyPI: pip install correctover
- Protocol spec: CCS v1.0 (5-dimensional verification framework)
This disclosure follows responsible disclosure principles:
- 30-day advance notice was not provided due to the public nature of API endpoints tested
- All tests were conducted using official, publicly available API endpoints
- No exploits, no unauthorized access, no data exfiltration
- All findings are reproducible using the methodology described above
- Full test logs available upon request to wangguigui@correctover.com
The Cryptographic Compliance Standard (CCS) is an open verification protocol for autonomous agent systems. It defines 5 mandatory verification dimensions:
1. Schema Validation — Output structure compliance
2. Cryptographic Provenance — Input/output traceability
3. Hallucination Detection — Factual accuracy verification
4. Drift Monitoring — Model behavior change detection
5. Cost/Token Auditing — Resource consumption tracking
CCS is provider-agnostic and does not require specific model dependencies.
*This document is released under CC BY 4.0. Attribution: Correctover, "Technical Disclosure: LLM Output Verification Failures," 2026-07-07.*