Summary
The LlamaIndex integration (qwed_sdk/llamaindex.py) and the CrewAI integration (qwed_sdk/crewai.py) label content as verified without running any verification:
QWEDQueryEnginereturnsVerifiedResponse(verified=True, status="PASSED")for every response that does not match one of five arithmetic regexes. No QWED engine is called. Withverify_facts=True(the default), a fact-style query (what is,who is,when did,where is,how many) takes a branch commented "For now, pass through" and also returnsverified=True.QWEDVerificationTransform(a node postprocessor) marks retrieved nodesqwed_verified=Truewhen verification raises._verify_contentends withexcept Exception: return True # Pass through on error. Theverify_codecall it wraps readsresult.verified, whichqwed_sdk.VerificationResultdoes not have (the field isis_verified). So every node that contains a fenced code block raises, is caught, and is marked verified.QWEDVerifiedAgent.verify_output(CrewAI) starts from{"verified": True}, then readsverification.verified. That raisesAttributeError, theexceptonly records the error, and the result keepsverified: True.verification_summary()then counts the output as verified. The same happens whenVerificationConfig(enabled=False), where no check runs at all.
Affected components
| Component | Location | Behaviour |
|---|---|---|
QWEDQueryEngine._verify_response |
qwed_sdk/llamaindex.py L148-169 |
Non-math and fact-style responses return {"verified": True, "status": "PASSED"} without a QWED call |
QWEDVerificationTransform._verify_content |
qwed_sdk/llamaindex.py L250-264 |
Any exception returns True; result.verified always raises AttributeError |
QWEDVerifiedAgent.verify_output (CrewAI) |
qwed_sdk/crewai.py L265-288 |
Default verified: True survives any exception; verification_summary() reports a 100% rate |
QWEDQueryEngine math path |
qwed_sdk/llamaindex.py L151-157 |
Also reads result.verified; this path happens to fail closed (ERROR) because of the same AttributeError |
Root cause
- The default branch of
_verify_responseis a pass-through, not a refusal. Content the integration cannot verify is reported as verified. _verify_content(LlamaIndex) andverify_output(CrewAI) treat an error as success.- The module reads
result.verifiedfromQWEDClient, whose result type exposesis_verified. This turns every code check into an exception.
Proof of concept (local, synthetic)
from qwed_sdk.llamaindex import QWEDQueryEngine
class FakeEngine:
def query(self, q, **kw):
return "The Eiffel Tower is in Berlin."
engine = QWEDQueryEngine(FakeEngine(), api_key="x")
r = engine.query("Where is the Eiffel Tower?")
print(r.verified, r.status) # True PASSED (no network call is made)
CrewAI, with the client returning an UNVERIFIABLE result (is_verified=False):
agent.verify_output("2 + 2 = 5") # {'verified': True, 'error': "... has no attribute 'verified'", ...}
agent.verification_summary() # {'total_outputs': 1, 'verified': 1, 'failed': 0, 'verification_rate': 1.0}
For the LlamaIndex postprocessor, any node whose text contains a fenced code block gets metadata["qwed_verified"] = True, whatever the code does, because self._client.verify_code(code).verified raises and the exception is swallowed as success.
Run against 7.3.0 with network access blocked: the query engine prints True PASSED, and _verify_content returns True for a node containing os.system(...) in a code block. The CrewAI case was also run against 7.3.0 with network access blocked (output above). main @ 4989490 has identical files.
Impact
Integrity of the verification signal. An application that uses VerifiedResponse.verified, the qwed_verified node metadata, or the CrewAI verify_output / verification_summary results to decide whether to show, store or act on a model answer will treat unverified, and possibly false, content as verified by QWED. Attacker capability: influence over the model output or the retrieved documents. No credentials are needed.
Affected versions
- The pass-through branches exist in tag
v2.0.0(the first PyPI release that shipsqwed_sdk/llamaindex.pyandqwed_sdk/crewai.py) through 7.3.0. Introduced in3ff1b4b. qwed 0.1.0does not contain either module.- The second copy of the integrations,
qwed_sdk/integrations/llamaindex.pyandqwed_sdk/integrations/crewai.py, has the same pass-through and attribute errors.
Patches
Fixed in 8.0.0 (ef6b5df):
QWEDQueryEnginereturnsverified=False, statusUNVERIFIABLE, with a reason, for any response no QWED engine verified. The pass-through branches are removed.QWEDVerificationTransform._verify_contentis verified only when a check ran and admitted; an error fails closed, and a node with nothing to check is markedqwed_verified: False, qwed_checked: False.QWEDVerifiedAgent.verify_output(CrewAI) starts fromverified: False; a disabled config reportsSKIPPEDand an errorERROR, both unverified.- The LlamaIndex, CrewAI and LangChain integrations read
is_verified, which follows the server's admission (GHSA-jppc-7r58-7pjh), instead of the missing.verified. - The same fixes apply to
qwed_sdk/integrations/.
Upgrade with pip install -U qwed (8.0.0 or later).
Workarounds
Do not act on VerifiedResponse.verified, qwed_verified node metadata or verify_output results. Call QWEDClient directly and gate on the server's admission == "ADMIT".
