2.4 — Test Knowledge Sources and Tools

Test Query Strategies for File Search

Effective test queries for retrieval validation:

1. Exact phrase query — paste a sentence verbatim from the uploaded document. This should always retrieve the correct chunk.
2. Paraphrase query — rephrase the concept in different words. Tests semantic matching quality.
3. Out-of-scope query — ask about something not in the documents. Confirms the agent says "I don't know" rather than hallucinating.
4. Cross-document query — ask a question that requires combining information from two different files.

After each query, expand the Tool calls section to see:

• Whether File Search was invoked.
• Which file and chunk were retrieved.
• The similarity score (if shown).

Reading the Tool Call Log

The tool call log exposes the raw mechanics of each agent turn:

Tool: file_search
Input:  { "query": "maximum upload size for vector store" }
Output: [
  { "file_id": "file-abc123", "filename": "overview.pdf", "score": 0.87,
    "text": "The single-file limit is 512 MB..." }
]

Citations appear inline in the response as numbered references (e.g., [1]). Each citation links to the source file and the specific chunk. If citations are absent when you expect them, it usually means the file search returned results but the model chose not to cite them — check the system prompt to see if citation instructions are present.

Common Issues and Fixes

Systematic Retrieval Debugging Steps

1. Check the Files tab — confirm status is Indexed (not Processing or Failed).
2. Submit an exact phrase from the document. If this fails, the file may not have extractable text.
3. Submit a rephrased version. If only exact phrases work, the embedding quality may be low.
4. Lower ranking_threshold to 0 and re-test — this reveals whether results exist but are being filtered out.
5. Increase max_num_results to 40 and check whether the correct chunk appears further down the list.
6. Review the chunk list in the Files tab to see how the document was split.

Validating Code Interpreter Results

After Code Interpreter runs, verify:

1. Correctness — manually compute a known value from the data and compare to the agent's output.
2. Code visibility — expand the tool call log to read the generated Python. Check for assumptions (e.g., column names, data types).
3. Error handling — submit malformed data or an ambiguous query. The model should report an error rather than hallucinate a result.
4. Chart output — if a chart was generated, download it and confirm axes, labels, and data ranges are accurate.