Using sentence-transformers/all-MiniLM-L6-v2 to Fix Our Web Crawler

Our web crawler was returning entire pages. A question like "what is the return policy" would get the whole product page back. A 5000 word article when we only needed 200 words.

This was wasteful. One page used 50,000 tokens. The model got confused reading all the noise and started making things up.

We used sentence-transformers/all-MiniLM-L6-v2 to filter. It converts text into numbers that capture meaning. Similar meanings land close together. So it finds chunks that match the question's meaning, not just keywords.

Result: 5,000 tokens per page instead of 50,000. 90 percent savings. No more hallucinations.

If you deal with web crawling or large text, this model is worth trying