Exposing BI code base to Claude

[u/WiseWeird6306]

I wanted to ask opinion of people on exposing everything a data team owes to Claude with a context layer explaining what is what with data lineage.

If a company has its own Claude subscriptionand then everything is exposed to Claude - report logic, metadata/dictionary of tables, logic/code of derived tables, operations of our pipelines etc via excel/markdown/html files explaning the whole context and then instructions Claude to not make assumptions/to ask verification questions (basically add safeguard rules in English in its instructions manual) then you expose this to end users company wide.

what is the issue with the idea? (if cost is not an issue)

[u/AutoModerator]

If this post doesn't follow the rules or isn't flaired correctly, please report it to the mods. Have more questions? Join our community Discord!

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[u/redleadereu]

The problem is you can't expect Claude to moderate itself. All those "don't touch these files", "you can't write to this schema" things you code into it are just a few clever prompts away from being bypassed.

You need enterprise level access control to make sure Claude follow the rules.

Then the other problem is; Claude is easier to teach than humans. "We apply a 0.5 multiplier here due to seasonality" is easy for Claude; it is not easy for ALL your users to grasp and remember.

AI is good for making the experts' job easier. It is horrible for giving everybody a capability to pull their own numbers.

Disclaimer: this all depends on the company and people. You can get it to work. That still would not mean you should allow people to calculate it, when they have a stake in the number being high/low.

[u/josemartinlopez → u/redleadereu]

"AI is good for making the experts' job easier. It is horrible for giving everybody a capability to pull their own numbers."

Amen

[u/WiseWeird6306 → u/redleadereu]

For some reason writing those statements like 'do not make any assumptions', 'do not make up tables or numbers' is working for us, and it has been obedient to follow the rules at pilot level. Is it still not a good practice? Am missing a point?

[u/BillEnough7863]

I work on a data team of 100+ at a company with thousands of employees, enterprise Claude subscription, the whole thing. The idea works. Getting it to work is a different question.

Even with genuinely good governance, the gap between "our context is documented" and "Claude reliably respects it" is enormous. You will lay out very explicit instructions and it will still do things it was told not to. You ask one basic question and it goes and does 27 other things, half of which it doesn't mention. We have caught plenty of that with guardrails, and the catch rate is improving, but without those protocols in place some of it would have been genuinely damaging. Then every model update shifts the behaviour again and you retune.

On your worry #1: the thing that kills company-wide access isn't the model, it's the users. The definitions you encode are the team's definitions. In normal conversation you find out constantly that people carry unspoken assumptions about what a metric means, and they don't know there's a definition to disagree with. Claude will answer the question they asked, confidently, in the semantics you gave it, and the person walks away with a number that doesn't mean what they think it means. That's the same failure mode self-serve BI has had for 20 years. This just makes the wrong answer arrive faster and better written.

On worry #2: I'd flip it. Curating the context layer, designing the access boundaries, and educating users on what the numbers actually mean is the job. It stops looking like writing queries. It doesn't stop being BI engineering.

One thing I'd push back on in this thread: English safeguard rules are a treadmill. If a rule matters, enforce it with permissions, not prompts. Read-only credentials, a narrow set of exposed models rather than the whole warehouse, scoped tools. Same work, but you build the boundary once instead of retuning instructions every release. It's also cheaper on tokens, which matters once you're off flat-rate pricing.

And the point nobody has a technical fix for: even if the system is flawless, you probably still don't want someone calculating a number they have a stake in being high or low. That's an org problem, not a context layer problem.

Might be different at a small company. At our scale, true self-service is still a ways off.

[u/WiseWeird6306 → u/BillEnough7863]

can you elaborate when you say 'the gap between "our context is documented" and "Claude reliably respects it"?

cause with your example where if we ask one simple question to Claude and it will do 27 different things, it was doing that but then we added rules that 'do this' 'do not do that' and now it is spitting the results in concise manner. So like we continuously putting in guardrails. I find it annoying but is this the way to go about it? Are we missing something in from long run perspective?

[u/Figdiggles27]

I just posted a file, and said in the prompt “using the file I just attached today” and it went ahead and used 500 tokens on yesterday’s file. I asked what happened when I said “use today’s attached file” and Claude says “sorry I should’ve used the attached file as directed but ai defaulted to yesterday’s file”, so good luck with that

[u/CautiousUse8597]

You've essentially described Genie's architecture, minus the enforcement layer. Worth looking at how it handles the parts that break.

Permissions: your plan has none. An instruction saying "don't reveal salary data" is a request, not a constraint, and once you expose this company-wide someone will get an answer they aren't cleared for. Genie runs generated SQL through Unity Catalog as the executing user, so row/column filters apply at query time. The model can't leak what the query can't return.

Staleness: markdown snapshots of table logic drift the moment someone ships a dbt change, and a confidently wrong stale definition is worse than nothing. Genie reads metadata from the catalog directly and mines query history and lineage to suggest updates, so the semantics track the warehouse instead of a doc someone wrote in March.

Retrieval, not stuffing: dumping the whole dictionary into context degrades accuracy, more irrelevant tables means more plausible-looking wrong joins. Genie scopes to domain spaces and retrieves only the relevant instructions, synonyms and join paths per question.

Verifiability: your version produces prose the user has to trust. Genie shows the SQL it ran, which is the actual audit surface. Plus certified metrics so "revenue" resolves the same way every time rather than being re-derived.

Measurement: benchmark sets of real questions, so you can tell whether an instruction change helped. Otherwise you're vibe-tuning a system a thousand people depend on.

Your instinct is right. The gap is that safeguards written in English aren't safeguards.

[u/WorldOfUmbro → u/CautiousUse8597]

+1 on this, Genie is the way to go for AI on enterprise data. You can use Genie MCP in Claude if you still want to use Claude

[u/Due-Association9901]

You're basically giving Claude the keys to the whole data warehouse and hoping the safeguard prompts will hold up. In my experience users always find creative ways to bypass those rules no matter how strict you write them.

[u/Strict_Fondant8227]

Cost aside, the failure mode isn’t Claude “knowing too much” - it’s company-wide users asking questions the context layer can’t constrain!

What usually breaks:

- Permissions: lineage docs and pipeline code often imply how to rebuild PII joins. English “don’t assume / ask first” instructions get ignored under pressure. You need real ACLs on what each role can retrieve, not a system prompt.

- Metric authority: if report SQL and the dictionary disagree, Claudr will pick whichever fits the question. One governed metric definition (with owner + as-of date) has to win.

- Hallucinated lineage: it will invent table relationships that look plausible. Force citations back to the files you ingested, and treat missing citation as “I don’t know.”

- Write path: read-only is fine... the moment someone asks it to “just fix the dbt model,” you’re in change-management land.

Same class of problem as Copilot over-fetching raw tables with only READ - expose a curated semantic + lineage pack first, not the whole repo, then widen carefully.

[u/fazz21]

I just create my own semantic layer, cant guarantee to expose lots of things to claude. But whenever I do work with data, most of my question already covered by it.

I think it is enough

[u/gjr23]

Wouldn’t it make more sense to give Claude a user account with access guardrails and permissions? Worst case is you have it create a local database where it pulls what it cannot per a more structured deterministic workflow and then give Claude unfettered access to the child database?

[u/WiseWeird6306 → u/gjr23]

well so creating a sematic layer would do that, wouldn't it?

[u/WendlersEditor]

There are solutions for this, like snowflake cortex analyst. If you want to roll your own then you do need a semantic layer (like you describe) but you also have to strictly control the tooling Claude has access to (e.g., have a programmatic filter that limits it to essentially read-only SQL commands, no update/drop/etc..

[u/TheSchlapper]

This is currently in the works at companies with really strong data governance

It’s just making sure everyone and the bot are all on the same page, but this is entirely possible, if not easy with the new .pbip file types for PowerBI

[u/WiseWeird6306 → u/TheSchlapper]

what if you don't have a strong governance?

[u/TheSchlapper → u/WiseWeird6306]

Then work on that first, then all other types of data work will be actually possible to be relied on

[u/eldrugo85]

Done a smaller version of this: knowledge base behind an mcp server, read only tools plus a dedicated retrieval step, so it reads the metadata where it lives instead of an export. The exports are the part i'd worry about, they drift from the pipelines in a few weeks and nothing tells you. And the 'don't assume, ask' instruction is the first thing to break: with ambiguous lineage it just picks the plausible branch. How do you plan to keep the files in sync?

[u/WiseWeird6306 → u/eldrugo85]

why are they drifting from the pipeline? what do you exactly mean by that?

[u/eldrugo85 → u/WiseWeird6306]

Exports are a snapshot, the pipeline keeps moving. Someone renames a column or adds a filter in a model, and the export gets rebuilt when somebody remembers, not when the change lands. In between the file describes a grain that doesn't exist anymore. Nothing breaks, no test fails, the answers just come back confident and slightly off, and you catch it only if you already knew the right number

[u/soopirV]

I built something like this to help me respond to RFPs and IT Risk Assessments and never have I felt more safe from AI taking over than I have since working with it. It has all it needs from my source files but it constantly made assumptions and crossed model/features consistently. I’ve honed it to the point that I get accurate responses 99% now, but it took a lot of iteration.

[u/WiseWeird6306 → u/soopirV]

how long did it take you to reach 99% accuracy? and that you have it what how's the scope of your work changed?

[u/WiseWeird6306]

I have another question: Should the logic be in context layer or sematic layer? Like building golden queries, we are building that in context layer. Why or why not?

[u/Sad_Ad_1681]

Not sure what your worry is. We did set it up that way. We‘re using Claude with all traffic going through AWS Bedrock. It has access to the full context layer (DBT) and the warehouse through a unified, self built MCP. We use it to build models, expand documentation and for end user consumption for ad hoc questions. The access to DBT ensures high accuracy and reduces token spend.

It’s a constantly growing environment but having users and developers use the same ecosystem helps us tremendously to build the right experience

[u/WiseWeird6306 → u/Sad_Ad_1681]

I had 2 worries: 1. should we be giving all this context to end users- all employees in the company? 2. I am not sure what my role as BI engineer will be anymore if everything is exposed to them and executives control the reporting and insights.

[u/CHC-Disaster-1066 → u/WiseWeird6306]

You could build an abstraction layer in between all the raw context and files.

I wouldn't give everyone in the company access to the raw files. Just superusers or tech savvy folks. For the others, build an agent that can answer their questions like "what's the logic of report xyz".