How we built a SOTA search engine using PostgreSQL, pgvector, and Qwen3 embeddings [P]

[u/NielsRogge]

I wrote a technical breakdown of how search works on Papers with Code.

The system combines keyword and semantic search, which produced better results than either approach alone. The stack includes:

• PostgreSQL with pgvector
• Qwen3-Embedding-0.6B for text embeddings
• Hugging Face Jobs with an NVIDIA L4 for batch embedding generation
• Hugging Face Buckets for storing artifacts
• A live embedding model served through Hugging Face Inference Endpoints

The same infrastructure also powers the “related papers” recommendations shown on individual paper pages.

Full write-up: How Hugging Face Inference Endpoints, Jobs, and Buckets Power Search on Papers with Code

I’d be interested to hear how others are implementing hybrid search for research papers or similarly technical content.

Disclosure: I work at Hugging Face and on Papers with Code.

[u/abnormal_human]

What exactly makes this SOTA? It's been well known that combining FTS with semantic search is an improvement over either alone for over decade. Google incorporated word2vec and bert based approaches into search by 2020. Is there something novel here, because all I'm seeing is an ad for huggingface's cloud services.

[u/NielsRogge → u/abnormal_human]

Quite unbelievable how Reddit works. When I posted about reviving Papers with Code in this subreddit, it got the most upvotes, made me a top 1% poster, and more. The reception was good, and people liked what I built.

Now all of a sudden, everything I learn and share about building the website gets disclosed as an "ad," is not interesting, and is apparently already known for decades. My only goal here was to openly share how to build a good search system on top of Postgres, using SOTA embedding models like Qwen 3, and to point to the MTEB leaderboard, a good place to find the best embedding models, how to optionally improve it with a reranker, etc. Also, explaining the difference between batch embedding a huge corpus on a GPU vs. using an on-demand inference endpoint to serve online traffic. The entire blog is a free resource.

You might be surprised how many people in the industry don't know about hybrid search with reranking. They just apply an embedding API, and that's it. They don't know about pgvector or similar tools and use a duplicated vector database just to store embeddings. They don't know that open-source embedding models have been outperforming closed-source APIs for quite some time now.

I feel like the Reddit crowd is just a bunch of grumpy and know-it-all people, haha, I don't know another way to put it.

[u/Benlus → u/NielsRogge]

For what it's worth I'm sure the ML community at large appreciates what you do, fyi I approved your post manually (because our filters removed it due to being an imagepost) since I thought that 1) your contributions are consistently of high quality 2) I assumed the community would welcome the update. I also disagree with the user below stating that the example is "not working in your favor". When I type "diffusion" into a ML search engine I'm not interested in a German guy named "Fick" outlining the law of diffusion in 1855 but in precisely the DDPM paper and it's connections to Flow Matching and recent literature that get returned. Just my thoughts, thank you for posting.

[u/SirPitchalot]

This particular example is not working in your favour. A basic one word physics term finding an 11 year old paper in a niche field.

[u/dioenatosenzadenti → u/SirPitchalot]

Niche field? That paper started diffusion models lol

[u/SirPitchalot → u/dioenatosenzadenti]

Diffusion, the PDE that Fourier (yes that Fourier) introduced the underpinnings of in 1822 and which governs heat transfer, key aspects of fluid flow, all manner of regularizers & energy terms in computational physics, that was used in the Manhattan project plus more recently anisotropic diffusion in computer vision for the past roughly 30 years and finally, yes, also the fancy schmancy slop generators some associate it with today?

Literally any commercial electronics you own has used this for thermal modelling. Including the GPUs you need to run contemporary diffusion models.

So yeah “niche field”. It’s been around 11 years out of 204 that the concept has been an absolute powerhouse of scientific progress. That’s why this search engine sucks.

[u/dioenatosenzadenti → u/SirPitchalot]

This is like calling Deep learning niche because it uses matrix algebra hahahha

[u/SirPitchalot → u/dioenatosenzadenti]

Do you think that matrix algebra is “niche”? I sure hope not…

“Deep learning” does not rename a fundamental mathematical topic that’s been in widespread use in scientific literature for two centuries. “Diffusion” being subsumed into diffusion models does.

It’d be like search for “attention” and having only results for transformers come back rather than the definition of the word, results from cognitive psychology and so on. Or “reasoning” redirecting only to “reasoning models”.