I keep coming back to index ownership when a RAG answer cannot be reproduced. Serving, evaluation, re-embedding, and governance may begin with the same corpus, then create separate copies with different data snapshots, embedding models, metadata, or index versions. The answer still looks deterministic, but nobody can identify the exact retrieval state that produced it.

One option is to treat the logical index as a versioned data asset. Hot retrieval can run through a vector database like Milvus, while offline jobs attach other compute to the same lineage and produce a candidate artifact. A fixed query set evaluates that candidate before the data snapshot and index version move into serving together.

The benefit is traceability. The cost is coupling. Permissions and compatibility become part of the index contract, rollback must restore a matched pair of data and artifacts, and one bad promotion can affect every consumer of the shared lineage. Independent application copies limit that blast radius even when they make drift harder to diagnose.

My current view is that consolidation makes sense when several workflows already share one corpus and embedding semantics. Separate release schedules or incompatible metadata are a reasonable boundary for keeping independent indexes. For teams that consolidated, what first showed you that copy drift had become the larger operational problem?

I built aimake — an incremental build system for AI/ML pipelines, basically make for AI apps.

GitHub: https://github.com/arjun988/aimake

The idea is simple:

Dataset → Preprocess → Embeddings → Index → Prompt → Eval → Report

aimake builds a dependency graph, fingerprints inputs by content rather than timestamps, and only rebuilds steps whose inputs actually changed.

So if you change your prompt:

Before:
Dataset       ✓
Preprocess    ✓
Embeddings    ✓
Index         ✓
Prompt        ✗ changed
Eval          ✗
Report        ✗

After:
2 rebuilt · 5 reused

aimake plan     # see what would rebuild
aimake build    # only stale steps run
aimake explain  # see why a step needs rebuilding

It's not Airflow (orchestration) and it's not DVC (data versioning alone).

It's make for AI pipelines: dependency graph + content fingerprints + incremental builds + caching.

Shipped so far

• Content-hash fingerprints instead of mtime-based caching
• Incremental + parallel builds
• plan / build / explain CLI
• Experiment comparison + hyperparameter search
• S3 cache + Hugging Face / DVC / Docker / Ollama / W&B plugins

You can try it with:

pip install aimake

The main reason I built this is that AI pipelines are expensive to rerun. Changing a prompt shouldn't mean recomputing your dataset, embeddings, vector index, etc.

If you build RAG , evaluation or any AI pipelines, I'd love feedback: what's the most painful step you wish was cached?

And if you think the idea is useful, a ⭐ on GitHub would really help.

I am embarking on a side project, mainly to learn a few neat bits of tech that I haven't been using day to day as an engineer yet.

The context of the app to help you understand, is for Golf players to capture round structured data such as hole scores / clubs / distances / etc etc, aswell as a commentary of the shot of hole. The idea being that they will be able to query their own data retrospectively and during a round to help with decisions etc...

If the structured data for a shot might look like
Golf Club: X Golf Club
Hole: 1
OutOfBounds: yes/no
DistanceHit: 200yards
Etc:

Then the commentary for that shot may also look like
"Hit the fairway, didn't commit to the shot so came out low as I hit it thin".

All initially stored in a SQL db, but obviously I have two forms of data here ' I think '.

After riffing with Claude, it believes that I'd see no benefit in setting up an RAG style search here with a vector db and embeddings ( specifically for the shot/hole commentary ). Instead I'm better off just using an LLM to generate a SQL query and get a chunk of data from SQL, and then just loading all of this data , both structured and commentary, into the LLM context so that it can be asked questions such as:

"What club do I usually hit on this hole x "
"How often do I miss the fairway on hole 10"
"On windy days, do I usually hit a driver here or a 4 iron"

Its hard to say whether RAG would benefit me or I'm better of just padding the context with the data stored in my SQL db.

Most databases are designed to answer: "What is the value now?"

They can model a more awkward question too, but usually require additional machinery:

Who claimed what, when was it considered true, how certain were they, and what did we believe before it was corrected?

I built Positorium as an experimental embedded evidence database for that second kind of question. Rather than overwriting one claim with another, it preserves contradictory claims together with their sources, certainty, effective time, assertion time, corrections, and retractions.

It is not intended to replace PostgreSQL or another operational database. The idea is to use it as a focused evidence layer for things like compliance, investigations, conflicting master data, or any process where retaining the history of disagreement matters.

The new Python package embeds the Rust engine directly in the Python process, so there is no separate server. It supports both ephemeral in-memory databases and append-only persistent stores.

Install the beta with:

python -m pip install --pre positorium

A small example:

import positorium

with positorium.Database.memory() as database:
    result = database.execute_one(
        """
        add role organization, risk_assessment;

        add posit
          [{(+company, organization)}, "Northstar Trading", @NOW],
          [{(company, risk_assessment)}, "high risk", '2026-01-12'],
          [{(company, risk_assessment)}, "needs review", '2026-01-12'];

        search
          [{(?company, organization)}, ?organization, *],
          [{(?company, risk_assessment)}, ?assessment, *]
        return ?organization, ?assessment;
        """
    )

    for row in result.to_dicts(text=True):
        print(row)

This returns both assessments rather than choosing a winner or overwriting one of them.

Positorium is still an early beta and is intended for evaluation rather than production deployment. Wheels are available for CPython 3.9+ on Linux, macOS, and Windows.

If you have a small dataset where sources conflict or corrections matter, try the beta:

• PyPI package
• Source and documentation
• 60-second browser example

We spent a bit of time looking at this because our embedding line got bigger than our generation line once we started re-embedding nightly. Per-token pricing punishes re-indexing hard, iykwim.

Here are some notes on what we found, in case it saves someone the digging.

Staying managed (OpenAI / Cohere / Voyage): Simplest, quality's good, nothing to run. But it's per token, so cost scales with corpus size and every reindex. If your volume is low or spiky this is still the right answer, honestly. An idle GPU costs more than the API bill.

TEI (Hugging Face). Free, self-hosted, strong on embeddings and reranking. Main thing to know is it's one model per server, so a two-stage retrieve-then-rerank setup means running more than one deployment.

SIE (Superlinked, Apache 2.0). Comes in self-hosted and managed option, but embed and rerank come off one cluster, and it's OpenAI-compatible so existing code mostly just points at your own endpoint. Their published benchmark claims around 1/12 the cost at ~97% of hosted-API quality. Their numbers, so weigh accordingly. Managed version isn't live yet, so today it's self-host only.

The actual deciding factor for all of these was utilization. Self-hosting only wins once the GPU stays busy. We reindex nightly so ours does, but if I were low-volume I'd have stayed on the API and not thought about it again.

Curious what people running this in-house actually landed on, and roughly what token volume made it worth leaving the managed API.

We profiled LangChain in our RAG query path and ended up making it optional.

Wanted to share our experience because we weren’t using LangChain for agents, retrieval, or orchestration.

We were mainly using it as a common interface for calling different model providers. It gave us one abstraction across OpenAI, Anthropic, and other providers, which was genuinely useful while we were building and experimenting.

We didn’t initially set out to remove it.

We were profiling our RAG pipeline to understand where CPU was going. Looking at flamegraphs from individual query executions, we were surprised to see that roughly 15% of the CPU samples were attributed to LangChain-related frames.

That seemed high considering we were mostly using it as an abstraction around the model provider APIs.

So we tested a simpler path.

For OpenAI and Anthropic, we bypassed LangChain and called their SDKs directly. Everything else stayed the same: same retrieval pipeline, same prompts, same models, and same application-level behavior.

After the change, we measured roughly 10–12% lower CPU consumption per query in our workload.

There wasn’t one obvious massive bottleneck. The flamegraphs showed the overhead spread across a number of framework-level layers around the actual provider call, including things like callbacks, validation, object conversion, serialization, and additional call-stack depth.

Each cost was small on its own. Across every query, they added up.

We haven’t removed LangChain from the codebase.

Instead, we made it optional.

For OpenAI and Anthropic, which are our primary providers, we now use their SDKs directly.

For other providers, LangChain is still available as a common interface. That lets us keep the flexibility without putting the abstraction in the hot path of every request.

This isn’t really a “LangChain is bad” post.

LangChain was useful for us, especially earlier when we were experimenting with providers and wanted to move quickly. I’d probably make the same choice again.

The thing I’d do differently is profile the abstraction earlier.

We assumed the overhead of using LangChain mainly for model calls would be negligible. In our workload, it wasn’t.

If you’re running RAG at meaningful volume, it may be worth profiling a few representative queries and checking your own flamegraphs rather than assuming framework overhead is noise.

Would be curious if anyone else has compared direct OpenAI/Anthropic SDK calls with LangChain and what numbers you saw.

I have been a lawyer for 20 years. Before that, I was a LAMP stack web developer. I made web apps for small to medium sized businesses and some government working units. Tragically uncool, but PHP paid for some fine Top Ramen in law school.

I laugh that now I've made an equally uncool SQLite + PDF RAG app. But it works well for single users and small teams. I use PDF because in law you have to *correctly quote to the page, and everyone works with PDFs. It has 3 desktop apps (free on the Microsoft Store) and one optional paid SaaS tool for AI OCR and summarization.

Fact Extract Prep converts a folder tree to a flat folder of PDFs. OCR can be applied via Tesseract. Optional BYO AI corrects Tesseract if high accuracy matters. It piggybacks on the Tesseract text-to-image mapping because I ain't smart enough to figure out how to map that from scratch. Videos are converted to metadata and a frame every 10% of play time. Emails are opened and converted along with attachments, nested 5 layers. Batch jobs as needed and let it run. The amount of life this thing has given back to me and my staff...

Fact Extract Bookmarker splits big PDFs at the bookmark. If there are levels of bookmarks, you can pick the one you want to use. You can quickly page through a PDF and add bookmarks hitting the space bar. There is a cool BYO AI functionality that will add the bookmarks for you, and then you just adjust if/as needed. That took a while to get working, for me anyway.

Fact Extract Desktop is the main RAG tool. It ingests a folder of PDF files, chunks to the page, and optionally adds embeddings. Those are at the 1/2 page and full page chunking level for big ideas. SQL, thesaurus, and summaries for other searches. The app allows notes, collections, exports, cross-database searches. Exports can be text or PDF, and new PDFs can be assembled from existing pages/collections.

An MCP server in Desktop can be added by one click to Claude Desktop, OpenWork, Goose, and AnythingLLM. The MCP allows the AI to search, link directly to cited pages, export and rename pages, annotate, and save findings for future work. Databases have a global ID so they can be shared by users and the links still work. Think: Associate lawyer does discovery response review, and hands the senior lawyer a Word file with links to the Fact Extract database. The senior reviews and builds a deposition outline and exhibits. Whoot.

The SaaS reviews PDFs at the page, file, or detected document level. It does OCR with AI vision that far outperforms traditional OCR. It uses a "structure" of prompts to ask a user-defined set of questions of each chunk. The user gets that analysis as a spreadsheet and a Fact Extract Desktop database with the good OCR and the summary. On specialized topics, the summary facilitates review with an AI via the MCP. And since the MCP allows the AI to pull images as well as text, OCR or summary errors can be addressed in chat/agentic review. The SaaS accepts Fact Extract databases in lieu of PDFs. Summaries can be added or OCR reused. The price is lower since we don't have to OCR or detect documents.

You won't run a giant company or centralized app on this. But it works great for those small groups that don't need concurrent database writing access. And it's free. I only use the SaaS when I have to. Most of the time using a good AI is perfectly sufficient.

Let me say - many here build way more elegant solutions. I think this has something to add as a workhorse. I'm happy to discuss how I approached problems if anyone is interested. The database structure and structure specs are what I referred to as open.schema. They are available on the websites.

Fact Extract Desktop

https://apps.microsoft.com/detail/9mww2wn9lsvz?hl=en-US&gl=US

Fact Extract Prep

https://apps.microsoft.com/detail/9nm1vsbz26t4?hl=en-US&gl=US

Fact Extract Bookmarker

https://apps.microsoft.com/detail/9nkzp48qttf3?hl=en-US&gl=US

Tutorials

https://factextract.net/tutorials

Schema

https://factextract.net/specifications

I’ve been using GPT-4o Mini for a while, but I kept hearing about DeepSeek’s API being cheaper and faster. So I ran a side-by-side benchmark with real developer tasks. The results were closer than I expected in some areas, and very different in others.

I wrote up the full comparison here:

https://interconnectd.com/blog/280/deepseek-api-vs-gpt-4o-mini-2026-developer-technical-review/

Curious what others think who’ve tried both

We are building a company-wide knowledge base from scratch. The company currently has no ERP or other structured operational systems, so this database would become the first system of record.

It should store:

• hard operational data,
• employee tasks,
• project progress and decisions,
• completed work and current state,
• expectations and work results,
• process, purchasing, machine utilization, and workflow status.

The main inputs will be unstructured: employee emails, Teams conversations, and meeting transcripts with speaker diarization and employee identification. We may use LLMs for cleaning, classification, and extracting relevant facts.

The difficult part is knowledge quality. Information may be contradictory, outdated, or provided by people without sufficient expertise or decision-making authority. We already have a structured model of employee competencies and authority, so statements could be weighted accordingly. Each fact should probably retain its source, timestamp, validity period, confidence, and change history.

AI agents will use this knowledge base to monitor tasks, purchasing, workflows, machine utilization, process execution, and employee productivity.

The key requirement is that, as the volume of data grows, agents must still receive relevant, current, and trustworthy context.

What architecture and data model would you use here? Temporal knowledge graph, event sourcing, relational database with a semantic layer, or something else?

I've been working on Custodian Labs, a Python SDK for building and deploying LLM agents without having to separately wire up all the surrounding infrastructure.

Basic agent looks something like:

from custodian_labs import Custodian

agent = Custodian(
    model="gpt-4o",
    system_prompt="You are a helpful assistant..."
)

agent.deploy()

A few things I've added:

• Model agnostic: switch between different LLM providers without rebuilding your agent
• RAG built in: connect your own files/data sources
• Multi-agent support: build specialised agents that can work together
• Privacy/PII layer: the Guardian Layer can detect and protect sensitive data before it reaches the LLM
• Deployment handled: trying to cut down the amount of infra/config needed to get an agent running

The project actually started as just the privacy layer, but after getting feedback from developers we expanded it into more of an end-to-end agent SDK.

Would genuinely love feedback from other LLM devs:

What's currently the most annoying part of your agent stack?

And do you prefer abstractions like this, or would you rather have more direct control over each component?

GitHub:
https://github.com/Custodian-Labs/custodian-labs-python

Runnable Google Colab: simple agents, RAG + multi-agent examples:
https://colab.research.google.com/gist/SherryCodes123/065d3b67eab16bdca416836e0d39475a/simple-ai-agents-rag-multi-agents.ipynb

Has anyone built a traceable, high-recall “second brain”?
We’re working on a system that turns a large, messy archive — documents, notes, code, decisions, and historical versions — into useful and verifiable memory.
The problem we’re trying to solve goes beyond standard search or RAG.
We want the system to detect:
• duplicates and near-duplicates
• contradictions
• superseded information
• relationships between sources
• provenance behind every useful claim
…while minimizing the chance of missing relevant evidence.
The hardest tradeoff so far is coverage vs. reliability vs. cost.
We’re experimenting with things like sliced/partial reading, separate extraction and independent-review stages, mechanical validation, caching, and long-running workflows.
We’ve also started testing these ideas in shadow mode on real cases instead of relying only on isolated benchmarks.
I’d love to hear from anyone working on similar problems: high-recall RAG, e-discovery, systematic review, provenance-aware knowledge graphs, PKM/second brains, or long-running agent workflows.
A few things I’m especially curious about:
• How are you reducing cost without sacrificing recall?
• How do you represent contradictions and provenance?
• What do you automate vs. independently review?
• Which architectures actually held up once you moved beyond prototypes?
Happy to share what we’re learning as well. I’m particularly interested in comparing approaches with people who have already run into these problems at scale.

Disclosure: we build this. TekMyra is from LaconIQ, and I work there. We open-sourced the core on August 31, Apache-2.0, with the paper published the same day.

The RAG-specific problem we kept hitting is that the tokens you most want to drop and the tokens you cannot afford to lose look identical to a compressor optimising for ratio. Chunk boundaries, document ids, file paths, section numbers, dollar amounts, policy identifiers and citation markers are low-frequency, low-context, and highly compressible. They are also the entire basis on which a downstream answer can be traced back to a source.

TekMyra treats those as protected spans. Before anything is emitted, a verifier confirms each one is represented exactly once in the output. Not "present." Exactly once, because a duplicated identifier misattributes an answer as surely as a dropped one. If the check fails, the compressor retries on a safer route, and if the retry fails too, it raises and emits nothing.

From the README Numbers table, locked spans came out 68/68 on synthetic and 704/704 on long_context_v1, whose fixtures average 23,644 chars, which is roughly the shape of a real retrieved context window. On that corpus the reached-fixtures ratio is 0.2929, the share of content kept averaged over the fixtures the compressor actually reached, and the corpus-wide token reduction is 48.44%, with 14 refusals of 40 fixtures counted in the denominator as saving nothing.

Two honest limits, since this is the sub where they matter:

1. The exactly-once check is a presence-and-count check on spans that were classified as protected. It is a strong guarantee about identifiers and citations surviving the compressor. It is not a claim that the surrounding prose is semantically faithful, and we do not make that claim.
2. Refusals are the price. On the long-context corpus, 14 of 40 fixtures got no compression at all. Your retrieval path needs a pass-through branch for that case.

pip install tekmyra-core, Python 3.11+. Trained artifacts are deliberately not in the repo and ship as a separate release asset, so a clean checkout measures a no-op compressor until you fetch the bundle and check its digest. We would rather say that here than have you diagnose it from a flat ratio.

Repo: github.com/laconiq-ai/tekmyra

Paper: tekmyra.ai/tekmyra-paper.html

We also publish the comparisons where we come off worse. If you run it against your own retrieved corpora and the numbers disagree with ours, post them and we will look.

I’m starting to think the hardest RAG scaling problem is not query latency but index ownership. In a fragmented stack, the online retriever, offline evaluation jobs, re-embedding pipelines, and data-governance workflows can each create their own copy of the corpus or index. That makes it difficult to know which version produced a result and whether offline improvements ever reached serving.

My current view is that a logical index should be treated as a versioned data asset. Hot serving can still use a vector database like Milvus, while warm or cold workflows reuse the same index lineage through different compute modes. The important part is preserving the link between the data snapshot, embedding model, metadata, index version, and evaluation result.

The tradeoff is operational coupling. Sharing lineage reduces duplicate builds and drift, but publishing a new index now affects more consumers and needs an atomic promotion process. I would probably require offline validation against a fixed query set, then publish the data and index snapshot together so production never observes a half-built state.

For teams running both online RAG and offline corpus work, where do you keep the authoritative index lineage today? Would love to hear your thoughts.

Most RAG projects work great in testing and then quietly get worse in production, and the reason is usually that evaluation was never real to begin with, reading a few outputs and deciding it looks fine isn't measurement, it's confirmation bias.

There's a hands-on masterclass on Sep 12 covering how to fix that properly:

• A real eval harness combining deterministic checks and LLM-as-judge, not just one or the other
• Bootstrap confidence intervals and paired significance testing for model comparisons
• Evaluated RAG with retrieval metrics (recall@k, MRR), so retrieval quality is measured, not assumed
• Agents with guardrails and fallbacks that fail gracefully instead of compounding errors
• Full production observability, tracing, cost/latency monitoring, and a CI regression suite

Led by Bruno Gonçalves, PhD, founder of Data For Science, who trains engineers at Fortune 500 companies on this exact stack.

Link for more details

We have a multi tenant RAG path where ACL prefiltering works fine and recall still collapses. High cardinality metadata plus a stale ACL replica leaves too few candidates before ranking. Top k fills with generic public docs, the reranker confidently sorts them and the right private source never reaches generation. The citations look tidy and answer almost nothing which is honestly a painful failure mode.

I’m looking at Braintrust to inspect chunks and ACL metadata in traces, compare retrieval experiments, score groundedness and save failed queries as regression cases. I want recall at k by permission cohort and not just a final answer score.

How do you measure top k starvation when access filters run before vector search and do you overfetch safely or change the index layout?

Disclosure since it's relevant: I work at Schema Labs, one of the models below is ours. No link, just numbers.

Most table discussion here is about extraction, getting a clean table out of a PDF. There's a second problem that shows up after extraction and I've seen little written about it.

When you chunk and embed a table, most of the semantic signal comes from the header row. "customer_id, signup_date, monthly_revenue, churn_flag" does nearly all the work. Cell values contribute less than you'd expect. Fine on documentation tables written for humans. Less fine on real system exports where you get V1 through V57, or metric_14, or four-character codes from something decommissioned in 2011.

We measured how much accuracy lives in the headers. 20 numerical classification datasets from OpenML, each run twice, once as published and once with every column name stripped. Same splits.

Mean ROC-AUC with names removed:

Schema-2: 0.9230 (0.9230 with names, so flat)
TabuLa-8B: 0.8658
ConTextTab: 0.8541

Both others gave up roughly 7 points. Internal runs, not third-party replicated, datasets are public on OpenML if you want to rerun it.

Limitation someone will find anyway: stripping names doesn't strip position, and column order still carries signal on some of these. We didn't control for that.

What it means for a pipeline. If your embeddings mostly encode the header row, two tables with similar headers and totally different data sit near each other in vector space, and a table with garbage headers holding exactly what you want sits nowhere near the query. Not an extraction failure, so parsing tools won't catch it, and it won't show up in an eval using human-written test tables.

What seems to help, none of it novel: generate column descriptions once with a bigger model using sample values instead of headers, and embed that. Put null rate, cardinality and a few sample values in the chunk. Route numeric questions to SQL over a sidecar rather than hoping retrieval finds the row.

Curious if anyone's measured this on their own corpus. I suspect most people with mixed document and table sources never separated the two in evaluation, so header dependency just reads as "retrieval is a bit worse on the spreadsheets."

btw happy to share the full protocol and per-dataset breakdown if anyone wants to rerun it or pick holes in the method, just say so.

Most RAG tutorials stop at "chunk it, embed it, retrieve it." That works until your document set has versions — an amendment that overrides a clause, a policy update that supersedes an older one.

Here's the failure mode: your vector search retrieves both the old and new version with similar confidence scores, blends them into one answer, and you have no idea it just cited outdated information as current.

I spent months building a RAG pipeline for exactly this — legal/contract documents where "which version is current" matters as much as "what does it say." A few things that actually moved the needle:

- Running a knowledge graph alongside the vector store, specifically to track "this document amends that one" relationships

- Reciprocal Rank Fusion across vector + keyword + graph search instead of picking one

- A second LLM pass just for reranking — fusion combines scores, it doesn't understand content

- Two-pass generation: one pass to extract facts, a separate pass to flag what's missing (merging these into one prompt made the model quietly gloss over gaps)

Ended up writing up the whole architecture with the reasoning behind each decision, not just the diagram. Happy to answer questions on any of this in the comments.

I've recently extracted video scene data into BigQuery/SQL table and md format for exploration purposes.

I already have experience with BM25/vector semantic searchs before. It's just that I have focusing so much on the data pipeline and didn't have time to catch up with recent retrieval technique till last week.

This "MD" data and search through using harness tools seems to be trendy now. I was wondering is there any simple/quick recipe for building the agent loop myself as I can't ask the end user to use claude code. Looking to ship a minimal webapp fro demo purpose.

On top of my head it would be something like:

                            Tools
   ┌─────────┐      ┌──────────────────────┐      ┌─────────────┐
   │  Agent  │─────►│ Search · Find · Open │─────►│ OKF/MD Data │
   └─────────┘      └──────────────────────┘      └─────────────┘
        ▲                                                │
        │                                                │
        └──────────── loop while iter < max ─────────────┤
                                                         │
                                                    iter = max
                                                         │
                                                         ▼
                                             ┌──────────────────────┐
                                             │ Final Answer + Cites │
                                             └──────────────────────┘

Is there any standard practice with the tools setup like query reformulation or grep cmds ... etc?

Or I just have to install middleman and monitor the servers and see what harness are doing behind the back?
I was only able to find deadpan linkedin posts that keep repeating same useless info over and over.

Appreciate if anyone could share their experience if they ever done something similar before.

I plan to do a multimodal rag that performs the following:

1. Pdf extraction - text, images, charts, tables from pdfs using pymupdf4llm

2. Storing in qdrant, and metadata filtering

3. Hybrid retrieval

4. Reranker.

Here I am stuck at extraction phase itself.

I tried using pymupdf4llm to get the charts but it doesn't retrieve all the charts present. Any ideas?

Most RAG evaluation asks whether the right passages reached the model. I wanted

to measure what happens when they do and the model still can't answer — because

the answer is a relation *between* passages rather than a statement inside any

of them.

Setup: a five-document narrative corpus (260,204 words, 13,950 passages) and 38

questions asking whether event A precedes event B, where A and B are narrated in

different documents and share no character, place or causal link. No passage in

the corpus states either relation. Five models, one family (Qwen3, 0.6B to 14B).

Given the source passages as text, every model scored 0/38 and refused 92-100%

of the time. I think the refusal is correct — the ordering genuinely is not in

the text. Given the identical facts as a structured chronology block from an

explicit state store, an 8B model scored 28/38 (73.7%).

A four-condition ablation separates information from form. At 14B, form is

irrelevant: plain prose, sorted prose and a structured block all land at 73.7%.

At 8B, structure leads the best prose condition by 6 items (73.7% vs 57.9%).

So: an 8B model given structure matches a 14B model given prose.

Two controls I'd want to see if someone else posted this:

- Permuting the supplied story positions collapses accuracy to 10.5% (8B) and

21.1% (14B). The models follow the ordering they're given rather than

recalling the published text.

- A realistic retrieval baseline is also at the floor, and it fails by asserting

rather than refusing. Going from 4 passages to 32 drove refusal from 97% down

to 50% while accuracy stayed at chance. More context produced more confident

wrong answers.

Two things I got wrong, both found by auditing my own scorer and question

generator after v1 was already published:

1. v1 reported the 8B form effect as +32 points. A scorer defect wasunder-crediting the prose conditions. Corrected, the gap is 6 items, not 12 —roughly half what I claimed. Re-scoring 1,786 saved items produced 30 gainsand zero losses, so nothing published was inflated; two things wereunderstated, and correcting them shrank my own headline.
2. For 36 of the 38 questions, the gold answers derive from author-assignedstory positions rather than from evidence-backed relations, and thegenerator's own self-check recomputes the gold from the same rows. That checkis circular. So this benchmark measures agreement with an author-assignedordering — not whether a system reports what the evidence establishes.

That second one is the real limitation and it bounds what the paper can claim.

I've left v1 up rather than retracting it, with the corrections in §11.

Full write-up, including the two things the audit changed:

https://ai.bedvibe.studio/structure-not-scale/

Paper, data and code: https://doi.org/10.5281/zenodo.22169643

Happy to be told the 0/38 is a prompt artifact — I tried to kill it and couldn't,

but I'd rather find out from you than not find out.

Kept running into the same problem prepping documents for RAG: raw PDFs/scans/Word carry a ton of layout noise that gets embedded alongside the real content, and most "convert to text" tools either upload everything to a cloud API or skip structured JSON entirely.

Built Sygal to solve this for my own pipelines: local-only document conversion (PDF, Word, Excel, HTML, email, EPUB, etc. via MarkItDown), OCR for scans/images routed to whichever provider you pick (OpenAI/Claude/Mistral/NVIDIA/Gemini) with your own API key, nothing else leaves the machine.

Output is clean Markdown for chunking/embedding, plus structured JSON for the metadata your code needs (page, source, tables as blocks). Also a CLI (sygal convert) with strict JSON output and stable error codes, built to be scriptable in an agent pipeline rather than just a GUI tool.

More detail on the Markdown-vs-raw-file token cost and the RAG-prep workflow here if useful: https://sygal.app/blog/preparer-documents-pipeline-rag

Curious what others are doing for the "clean input" step before embedding, especially for scanned/legacy documents.

I’m curious what others are doing to manage RAG poisoning - I have a pipeline with multiple touch points with users able to introduce material - both through forms, document uploads and audio transcripts.

Have been looking at a multi layer approach of simple regex gates for common attacks and a second layer of a small model trained at spotting attacks. I’m trying to find a balance of effective enough without adding too much computational overhead. I already have a quarantine queue, so I can pass uncertain results to that.

Very interested in tactics others are using, and what types of attacks people have had to deal with.

GitHub: https://github.com/stevereiner/flexible-graphrag

Flexible GraphRAG v0.8.0 adds two more ingest pipelines — a Rust-based CocoIndex pipeline and a Visual Langflow mode — for three in total. Whichever one you configure, you keep the same configurable data sources and database targets, the same REST and MCP APIs, the same web UI, and the same .env configuration.

Architecture diagram: three ingest pipelines, one configuration

It also shows that the CocoIndex pipeline can run standalone through app.py and the CocoIndex CLI, without the FastAPI REST server.

What Flexible GraphRAG Provides

Flexible GraphRAG is an Apache-2.0 open-source AI context platform for document processing, knowledge-graph construction, hybrid retrieval, GraphRAG/RAG, and AI-assisted query/chat.

It supports Docling, LlamaParse, and LiteParse document processing; ontology/schema-aware knowledge-graph extraction; 13 LLM providers; and hybrid retrieval across full-text, vector, property-graph, and RDF/SPARQL backends.

It supports incremental updating of all target databases, using event change detectors for the 10 auto-sync data sources — either with the original Python-based / PostgreSQL-managed incremental update system (default and Langflow pipelines), or with the Rust-based CocoIndex engine (CocoIndex pipeline).

The main backend is Python, with full support for LlamaIndex and LangChain — and now CocoIndex "native" too. Angular, React, and Vue TypeScript front ends are included, together with an MCP server.

Three Ingest Pipelines — Pick One

The existing Python-based Flexible GraphRAG pipeline remains the default. You configure one of the three:

• Default pipeline: LlamaIndex / LangChain ingest, hybrid search, AI query/chat, and Python/PostgreSQL-managed incremental updates.
• CocoIndex pipeline: Rust-based incremental processing; can mix CocoIndex-native and Flexible GraphRAG components.
• Langflow flows: customizable visual ingest/search/AI-query flows with 12 Flexible GraphRAG Langflow components.

Important: CocoIndex mode and Langflow mode are separate modes; they cannot be enabled together.

CocoIndex Integration

CocoIndex: https://github.com/cocoindex-io/cocoindex

The CocoIndex pipeline works within Flexible GraphRAG and can use the same UI, REST APIs, MCP APIs, data source configuration, and Flexible GraphRAG targets as the default pipeline.

It can mix:

• CocoIndex-native components: source connectors, functions, splitting, and CocoIndex-native graph/vector target connectors.
• Flexible GraphRAG components: data sources, LlamaIndex/LangChain targets, LiteParse/Docling/LlamaParse document processing, splitting/chunking, ontologies, and knowledge-graph auto-building extraction.

For each configured backend category—source, chunker/splitter, property graph, vector database, search backend, and KG extractor—the .env configuration can select llamaindex, langchain, or cocoindex. The actual database selection is configured independently.

Incremental Processing

In CocoIndex mode, Rust based CocoIndex provides the incremental update engine instead of the default Flexible GraphRAG Python/PostgreSQL per-file-state auto update incremental system. PostgreSQL remains available to track the multiple data sources configured through the UI.

For Flexible GraphRAG data sources used by the CocoIndex pipeline, the existing event change detectors continue to be used. These include:

• Alfresco ActiveMQ
• Nuxeo Kafka
• Amazon S3 SQS
• Azure Blob change feed
• Google Cloud Storage Pub/Sub
• Google Drive Changes API polling
• OneDrive/SharePoint Microsoft Graph delta queries
• Box Events API polling
• Local filesystem watchdog

Use the Flexible GraphRAG CocoIndex Pipeline Outside the UI App Too

The CocoIndex pipeline's app.py can also be used outside the UI application, for custom mixed applications that combine CocoIndex-native and Flexible GraphRAG components in your own code.

CocoIndex CLI support is available as well, so the same pipeline can be run standalone — without the FastAPI REST server or any of the web front ends.

Langflow Integration

The Langflow integration enables visual flows for ingest, hybrid search, AI query, and AI chat behind the Flexible GraphRAG UI, REST API, and MCP server.

The supplied flows reproduce the default pipeline behavior but can be visually customized. The integration includes 12 configurable Flexible GraphRAG Langflow components that can also be used in other applications.

The components are themselves Python-based, and use the Flexible GraphRAG Python "framework" — the same code the default pipeline runs. So this is not a separate reimplementation: it makes the default Python-based pipeline (hybrid_system.py) modular and visually customizable.

Langflow plus the components can run in a separate virtual environment, or through the Flexible GraphRAG backend Docker image together with the Langflow + Flexible components image.

When ENABLE_LANGFLOW_FLOWS=true, the app UI, MCP server, and REST API use the visual flows. All 14 data sources and the selected document processor—Docling, LlamaParse, or LiteParse—are supported. If ENABLE_INCREMENTAL_UPDATES=true is also enabled, changes from the auto-sync sources run through the Langflow ingest flow.

Sources and Targets

• 14 data sources, with 10 auto-sync sources: Alfresco, Nuxeo, Amazon S3, Google Cloud Storage (GCS), Azure Blob Storage, SharePoint, OneDrive, Google Drive, Box, and local filesystem. Other sources are CMIS, web pages, YouTube, and Wikipedia.
• 15 property-graph databases: Neo4j, ArcadeDB, FalkorDB, LadybugDB, Amazon Neptune, Neptune Analytics, Memgraph, NebulaGraph, Google Cloud Spanner, ArangoDB, Apache AGE, HugeGraph, SurrealDB, TigerGraph, and Azure Cosmos DB Gremlin.
• 4 RDF/triple stores: Apache Jena Fuseki, Graphwise/Ontotext GraphDB, Oxigraph, and Amazon Neptune RDF.
• 10 vector databases: Qdrant, Neo4j, Elasticsearch, OpenSearch, Chroma, Milvus, Weaviate, Pinecone, PostgreSQL/pgvector, and LanceDB.
• 3 search engines: OpenSearch, Elasticsearch, and BM25.
• 13 LLM providers: OpenAI, Ollama, Azure OpenAI, Google Gemini, Anthropic Claude, Google Vertex AI, Amazon Bedrock, Groq, Fireworks AI, OpenAI-compatible endpoints (LM Studio, vLLM, LocalAI), OpenRouter (200+ models), LiteLLM Proxy (100+ providers), and vLLM.

Databases and dashboards can be enabled from the Docker Compose configuration. Optional Docker images are available for the backend, Langflow plus Flexible components, and React/Angular/Vue front ends:

https://hub.docker.com/u/integratedsemantics

Also Since v0.6.3

v0.7.2

• Added Nuxeo as a document/content data source alongside Alfresco.
• Added OAuth 2.0 support for Nuxeo, Alfresco, and MCP.

v0.7.1

• Added LiteParse document processing alongside Docling and LlamaParse.
• Delivered Langflow integration fixes and an optional Langflow Docker image bundling the 12 Flexible components.
• Added Microsoft Graph delta-query support for more efficient SharePoint and OneDrive incremental updates.

Earlier Announcement

Previous v0.6.3 Reddit post:

https://www.reddit.com/r/Rag/comments/1ucummg/flexible_graphrag_v063_available/

Feedback, issues, ideas, and PR contributions are welcome.

Hey everyone. I open-sourced RegX - a production-ready boilerplate for building modular Retrieval-Augmented Generation (RAG) pipelines.

I built this to skip the boilerplate setup phase when creating LLM apps. It handles document ingestion, async background processing, and chat interfaces out of the box so you can just plug in your data and start testing.

The Stack: Python, FastAPI, Weaviate, MongoDB, Redis + Celery, and Streamlit.

What it actually does:

• Modular LLMs: Swap between OpenAI, Gemini, and Anthropic using a Factory Pattern just by changing the .env file.
• Async Data Ingestion: Markdown documents are chunked (preserving headers) and ingested into Weaviate in the background using Celery and Redis, without blocking the API or UI.
• Embeddable JS Widget: It comes with a native ragx-widget.js script. You can drop it into any standard HTML page to instantly overlay a chat interface connected to your FastAPI backend.
• Chat History: Session-based history tracking stored in MongoDB.
• Observability: Native hooks for Langfuse/LangSmith tracing and Sentry error tracking.
• Fully Dockerized: The entire architecture (API, UI, Workers, DBs) spins up with a single docker-compose up --build.

Repo: https://github.com/arch11110/ragx
Demo: https://www.youtube.com/watch?v=qdTqpSZrATY

Hey, I’m curious how people here handle this in practice.

A RAG system, or any similar system, is only useful if the data behind it is actually correct. So how do you make sure it is?

Do you have a specific process or solution for this? Are you using any tools, or have you built something yourselves? What does this look like in your setup?

Would love to hear how people are actually doing this.