Semantic relevance is not the same as sufficient evidence.
A 10-K contains risk language across multiple sections, not only the dedicated risk-factors section. Semantic retrieval can surface potentially relevant passages, but a close match to a query does not establish that the passage supports a specific risk finding.
The project asked how an analyst-facing RAG system could preserve that distinction. Success meant returning traceable evidence under a declared standard—and returning no finding when the retrieved text did not meet it.
My contribution
I built the full prototype: SEC EDGAR acquisition and text cleaning, chunking and embeddings, local vector retrieval, category-specific search queries, three validation modes, cross-mode comparison, a Streamlit interface, and structured exports.
The work focused on how the system makes and communicates a judgment, rather than on presenting generic document summarization as analysis.
Four choices made the reasoning process inspectable.
Search the full filing
The system ingests the complete 10-K because financial, operational, and regulatory evidence can appear outside Item 1A.
Keep retrieval and classification separate
Twelve queries across three risk categories generate a broad candidate pool. A separate validator decides whether each candidate meets the active evidence standard.
Make the threshold explicit
Aggressive, balanced, and conservative modes show how results change when the definition of sufficient evidence becomes stricter.
Allow an empty result
Each mode can return an empty array when no candidate qualifies. The interface presents absence of evidence as a valid outcome rather than forcing a generated answer.
The output preserves the path back to evidence.
- Risk summary
- A concise description of the supported finding.
- Source chunk ID
- A deterministic reference back to the filing passage used as evidence.
- Key evidence
- A short excerpt selected from the retrieved chunk.
- Threshold rationale
- An inspectable explanation of why the passage was classified under that mode.
- Cross-mode status
- A comparison of whether the finding appears under one, two, or all three standards.
A generated rationale improves inspectability, but it is not proof that the model’s judgment is correct. The source passage remains the primary object of analyst review.
What the prototype demonstrates—and what it does not
The working application demonstrates full-document ingestion, multi-query retrieval, evidence-linked structured output, threshold comparison, explicit refusal behavior, and exportable findings.
The project does not yet include a labeled benchmark, systematic retrieval metrics, faithfulness scoring, or user study. I therefore treat its value as evidence of system design and implementation, not evidence that it achieves production-level accuracy.
Next validation steps
- Create a filing-level benchmark with analyst-reviewed relevant passages and risk findings.
- Measure retrieval recall separately from classification precision and source faithfulness.
- Test whether the three modes produce meaningfully different review burdens for analysts.
- Replace summary-overlap matching with a more robust finding-resolution method.
- Evaluate cost and latency under a realistic document and company workload.