Update from the FreshCtx maintainer: FreshCtx now protects the same pre-action freshness boundary across Agno, LangGraph, and the OpenAI Agents SDK.

The framework changes, but the failure mode remains the same: 1. An agent observes a file, API response, database record, approval, or other dependency. 2. It reasons from that evidence. 3. The dependency changes before the tool or action executes. 4. FreshCtx revalidates the declared evidence at the action boundary and returns CURRENT, STALE_SOURCE, STALE_REASONING, or UNVERIFIABLE.

Recent releases added: - Agno 2.9 pre-tool integration - LangGraph protected-node integration - OpenAI Agents SDK tool-boundary integration - A shared experimental pre-action contract - Async and bounded concurrent validation - Validation budgets and audit evidence - Semantic policy/configuration validation

The new policy example also distinguishes between raw-file changes and material field changes. A descriptive edit can remain CURRENT, while a change to a declared policy limit becomes STALE_SOURCE. Invalid or incomplete configuration becomes UNVERIFIABLE.

FreshCtx 0.7.0 is open source: https://github.com/Hyperwise-LLC/freshctx

I maintain the project. I’m looking for one specific kind of feedback: does the pre-action contract map naturally to where your framework executes tools, or would it force you to restructure the workflow?

I've been learning more about building production-ready AI agents with LangChain, and one thing that stood out to me is that giving an agent access to tools creates a very different problem than simply generating text.

For example, an agent might have access to:

- search()

- sendEmail()

- deleteUser()

- makePayment()

The question becomes: how do we make sure the agent only performs actions that it's actually allowed to perform?

I recently went through LangChain's middleware-based guardrail approach and wrote up a practical guide covering:

- Deterministic vs model-based guardrails

- PII detection and redaction

- Human-in-the-loop approval for sensitive tools

- beforeAgent guardrails

- afterAgent guardrails

- Custom middleware

- Combining multiple guardrails in one agent

One pattern I found particularly useful is:

User Request

↓

Input Guardrail

↓

PII Protection

↓

Agent

↓

Tool Call

↓

Human Approval (if required)

↓

Execute

↓

Output Guardrail

↓

Final Response

I also included TypeScript examples showing how these middleware components can be implemented.

I wrote the full guide here:

https://medium.com/@nayankunwar678/guardrails-in-langchain-a-practical-guide-to-building-safe-ai-agents-68a9d5783c9e

I'm curious how others are approaching this in real projects.

Do you mainly use:

1. LangChain's built-in middleware?

2. Custom middleware?

3. External guardrail systems?

4. A combination of these?

And where do you usually put your most important checks — before the agent, around tool calls, or after the agent?

Solo dev here. I've been bothered for a while by how expensive it gets to do GraphRAG ingestion with LLM extraction when you have real volume — you pay per token, it's slow, and sometimes the JSON comes back broken and you retry.

So I built a small non-autoregressive decoder (~37M params) that takes a sentence embedding (SONAR, Meta's multilingual space) and outputs the knowledge graph directly: entities + typed relations in a single pass. No text generation anywhere, so malformed output is structurally impossible.

The LlamaIndex part: there's a connector class, three lines to production:

extractor = CogitoGraphExtractor(checkpoints, "vocab-prose.json")
triples = extractor.extract(node.text)
extractor.to_neo4j(driver, triples, source=node.id_)

It takes plain text (whatever your pipeline hands it), returns string triples, and maps them to Cypher MERGE with per-edge provenance. There's also extract_batch so ingestion does one encoder call for N chunks.

Honest numbers, all on held-out data with the splits documented in the repo: tool-call structures 1.000 F1, python code 0.781, prose with entity candidates 0.827, open-vocab prose 0.651. Prose is the hard modality — the failed attempts (focal loss, LLM label distillation, char-level generation) are all in the changelog because I think negative results are half the value.

Everything was trained on a single RTX 5070 at home. Decoder heads are Apache-2.0; heads-up that the SONAR encoder itself is Meta's CC-BY-NC, migrating to BGE-M3 is next on the roadmap for a fully commercial-clean stack.

Repo: https://github.com/DeliVali/cogito-estella
Weights: https://huggingface.co/DeliVali/cogito-estella

Where I'd really value feedback (building this solo, so outside eyes matter a lot):

1. The candidate design: for prose, the extractor picks relations between entity candidates that come from the text + your existing graph nodes. Does that fit how your pipelines actually work, or do you need fully open extraction even at lower accuracy? (0.827 with candidates vs 0.651 open-vocab — that's the tradeoff)
2. The relation vocabulary is 60 verbs. When the sentence's verb isn't there, the model picks the nearest one ("approved" → "support"). Is that acceptable degradation for your use case, or is exact relation wording a dealbreaker?
3. What would make you actually try it? Be brutal — missing docs, the fairseq2 dependency, no async API, whatever. The friction you name is what I fix first.
4. If you run GraphRAG ingestion today: what does extraction cost you per 1M chunks, roughly? I want to check if my "1000× cheaper" math survives contact with real setups.

If you try it and it breaks, open an issue and I'll fix it fast — early issues are gold for me. And if anyone wants to benchmark it against their current extraction stack, I'll happily help set it up.

I spent about 6 weeks building an agent that drives our desktop app for regression testing. tools for screenshot, click and type, a vision model for grounding, a loop that keeps going until the goal is met. it demos quite well but then i ran the same 20 flows 10 times each and got roughly 12 percent variance in outcomes with nothing changed between runs. The agent sometimes takes a different route to the same end state, which is okay for an assistant and useless for a regression gate where the entire point is that identical input produces identical output. i'd built something that can operate the app but can't tell me whether the app changed.

Before I spend another 6 weeks on determinism i'd like to know whether i'm rebuilding something that exists. Bytebot and Goose are both further along than mine on the driving side and i assume they hit the same wall, but i can't find anyone writing about what they did after that. The dedicated QA models like Askui and Eggplant appear to solve it by making you write the steps explicitly and only using the model for perception.The specific thing i'm stuck on is whether you can get determinism out of an agent loop at all or whether the answer is to remove the loop.

Has anyone got an eval setup for this that isn't just running it 10 times and eyeballing the diffs?

Hi community, I am so confused . I have learnt ReAct loops , prompt engineering, Rag , Vercel AI Sdk few guadrails from the first principals and built minimal terminal agents .

Should I go to Langchain/LangGraph next or should I built some solid project from the things I have learnt and then move to next .

What projects should I build?

Maintainer here. I built a small LangChain integration that lets an agent send and track faxes, and put it on PyPI as langchain-ictfax. Fax tooling is pretty thin in the LangChain ecosystem, so I figured it was worth sharing, and I would like a sanity check on how I shaped it.

What it gives an agent: - upload_fax_document: upload a PDF, TIFF or image and get a document id back - send_fax: send that document to a number and return a transmission id - get_fax_status: poll delivery status - list_faxes: list transmissions with their status

All four are bundled by an ICTFaxToolkit, so you pull them into an agent in a couple of lines:

from langchain_ictfax import ICTFaxToolkit
tools = ICTFaxToolkit.from_credentials(base_url=..., username=..., password=...).get_tools()

Being upfront: it is a client for an ICTFax server, so it needs a reachable ICTFax or ICTCore install and an API account to actually send. ICTFax itself is open source. I am not trying to sell anything here, what I am after is feedback on the LangChain side: does the toolkit shape read well, are the tool descriptions clear enough for an agent to pick the right one, and is returning raw ids the right call for tool outputs or would you expect richer objects?

pip install langchain-ictfax Repo: https://github.com/ictinnovations/langchain-ictfax

Happy to answer anything about the wiring.

At some point, you've built this: a review UI that shouldn't have existed. Ours was a Retool app, stitched together on a weekend, because the extraction tool we were using handed us a confidence score and nothing else. No reason it was low. No way to route it. No path back into the pipeline once someone fixed it.

The problem we kept running into

A single confidence number isn't a workflow. It's a data point with nowhere to go. Most document AI tools stop there, so the developer ends up building the rest: a queue, a UI, some way to route flagged fields to a human, and a script to patch the correction back into the record because nothing does that automatically.

By the time you're done, you've built a second product just to make the first one usable.

Where the gap actually comes from

Review gets treated as an afterthought, something bolted on after extraction instead of built into the pipeline. So when a field comes out wrong, there's no context for why, no distinction between "the value is genuinely uncertain" and "the value depends on someone external," and no record of what changed if you correct it.

How we tried to close it with IDPForge

Flagged documents land in a queue, but they come with a reason attached. A trigger chip tells you exactly why the document is there, low confidence on a specific field, or a validation failure like line items not summing to the total. You're only shown the fields that were actually flagged. Everything that already passed stays untouched.

When you correct a value, the original isn't overwritten. It sits alongside your correction in a field ledger, so there's a record of what the model got wrong and what a human said instead.

Not every flagged field has an answer sitting on the page. Sometimes you're waiting on a vendor to confirm a PO number, or finance to approve an unbudgeted line. For that, there's Park, a separate action from correcting, so you're not tempted to guess just to clear your queue. Parking stops the clock on that document too, so it doesn't quietly wreck your team's handling-time numbers.

Submit a document, and it either delivers straight to its destination or goes through a verifier first if a coverage rule says it should. Either way, the correction doesn't vanish into a database somewhere. It's part of the record.

And every correction matters beyond that one document. We're already working on tightening the loop between what your reviewers fix and what the model does next time. That's not live yet, but it's where this is headed.

That Retool app we built years ago, this is basically it. Except it already exists, it's part of the platform, and nobody had to spend a weekend on it.

Has anyone built a traceable, high-recall “second brain”?
We’re working on a system that turns a large, messy archive — documents, notes, code, decisions, and historical versions — into useful and verifiable memory.
The problem we’re trying to solve goes beyond standard search or RAG.
We want the system to detect:
• duplicates and near-duplicates
• contradictions
• superseded information
• relationships between sources
• provenance behind every useful claim
…while minimizing the chance of missing relevant evidence.
The hardest tradeoff so far is coverage vs. reliability vs. cost.
We’re experimenting with things like sliced/partial reading, separate extraction and independent-review stages, mechanical validation, caching, and long-running workflows.
We’ve also started testing these ideas in shadow mode on real cases instead of relying only on isolated benchmarks.
I’d love to hear from anyone working on similar problems: high-recall RAG, e-discovery, systematic review, provenance-aware knowledge graphs, PKM/second brains, or long-running agent workflows.
A few things I’m especially curious about:
• How are you reducing cost without sacrificing recall?
• How do you represent contradictions and provenance?
• What do you automate vs. independently review?
• Which architectures actually held up once you moved beyond prototypes?
Happy to share what we’re learning as well. I’m particularly interested in comparing approaches with people who have already run into these problems at scale.

If you are building RAG pipelines, agents, or data-extraction tools, you probably inject API responses or database rows directly into your LLM's context window. The problem? JSON is the standard for APIs, but it is notoriously terrible for LLMs.

You end up paying for thousands of useless structural tokens ({, ", ,, \n) which increases latency, drives up API costs, and eats into your context window limit.

I got tired of this and built two pure Python, zero-dependency micro-tools to compress structured data before it hits the LLM.

1. json-to-yaml-lite (The General Fix)

It’s a known trick that LLMs understand YAML just as well as JSON, but YAML consumes about 20-30% fewer tokens because it drops the quotes and brackets. However, standard libraries like PyYAML are massive, require C-bindings, and slow down serverless cold starts (AWS Lambda).

I built a purely AST-based micro-converter: * Token Efficient: Strips all unnecessary syntax while safely escaping edge cases (like strings with colons/newlines). * Zero Bloat: No external dependencies. Drops right into your pipeline. * Repo: Encephos/json-to-yaml-lite

2. json-to-toon-lite (The Heavy Compressor)

YAML is great, but if you are injecting an array of similar objects (e.g., 50 search results or users), repeating the keys every single time is still a massive waste.

TOON (Token-Oriented Object Notation) solves this by detecting uniform arrays and compressing them into a highly dense, CSV-like tabular format. * Massive Savings: Compresses uniform arrays like [{"id": 1, "name": "A"}, {"id": 2, "name": "B"}] into [2]{id,name}: 1,A | 2,B (saving up to 60% of tokens). * Safe Fallbacks: If the objects in the array have varying keys, it gracefully falls back to standard YAML bullet formatting. * Pure Stdlib: Again, zero dependencies. Just pure Python logic. * Repo: Encephos/json-to-toon-lite

Both tools are designed for devs who want to optimize their LLM API costs without pulling in massive frameworks. I’d love to hear your thoughts on data serialization for LLMs!

I have been trying a bunch of harness models and frameworks for the last month, and I realize most agent frameworks want you to rebuild your runtime around their harness.

We went the other way with AURA Harness: a thin membrane around loops you already run, plain Python, Ollama, LangGraph, whatever. You keep the body/runtime, while AURA records and optionally gates what crosses the boundary. Shouldn't this be what harness is about??

Local-first by default:

* Works offline with `agent_ref` \+ session IDs, no cloud identity required * `integrations/ollama/llama_loop.py,` stdlib HTTP against Ollama (`llama3.2:1b` is our dev default) * Optional verified operator identity if you need it later, not required for OSS/dev

What you get on close: JSONL spine, audit report, hash chain, `aura verify chain` for CI.

Loose coat = audit-only logging. Tight coat = rules/gates at egress when you wire tool paths.

Open source (Python): [github.com/ARPAHLS/aura](https://github.com/ARPAHLS/aura)

Happy to answer setup questions, especially Ollama related + “wrap my script, don’t replace it.”

Contributors more than just welcome, bunch of good first issues open <3

I built a React agent that executes a few tasks, such as executing a skill or tool and providing an answer. It’s connected to RAG. Now, if a user asks a query about where to go to RAG and answer it, that same single question takes about 40 seconds. I wonder why I need to send the full prompt when it only needs to hit the RAG pipeline. I need some help fixing this. If some of you are considering having a sub-agent, I think a sub-agent creates a split-brain problem, but it improves other things. Any comments or help would be appreciated.

Genuine question, I keep going back and forth on this.

Agent calls a SQL tool. Query comes back with 400 rows. Obviously you don't put 400 rows in context. So what do you put.

What I'm doing right now is dumb. Truncate to the first 20 rows and a row count. It works for "how many customers churned" and falls apart the second the question needs anything about the shape of the result, because the model has no idea whether those 20 rows are representative or whether the interesting stuff is in row 300.

Tried summarising the result with a second call. Better answers, but now every tool call is two model calls and the latency doubled on a step that used to be fast.

The other thing that bites is column names. If the query returns something like val_b or flag3, the model will confidently interpret it as whatever seems plausible from the question. It doesn't ask. It doesn't flag it. It just decides.

So:

Are you passing raw rows, a computed summary, or some schema-plus-sample hybrid? Does anyone compute stats server-side and return those instead of rows? And has anyone found a way to make the agent say "I don't know what this column is" rather than guessing, without stuffing a data dictionary into every prompt?

It's Tuesday. You're integrating a new document type into your pipeline. By lunch, your Postman collection has four different auth headers, three different pagination styles, and one endpoint that hands you back snake_case while another insists on camelCase.

Nobody warns you about this part.

The problem we kept running into

Document automation isn't one step; it's four: parse the document, split and classify it, extract the fields you actually care about, and clean up what comes out the other end. Most tools out there are genuinely good at one of these. Maybe extraction. Maybe parsing. That's exactly why developers reach for them, and it's the right instinct.

The trouble shows up later. Once that one stage is wired in, you still need something for the rest of the pipeline. So you bring in another tool. Then another. Now you're not building a document pipeline, you're building a translation layer between three vendors who've never heard of each other, each with their own idea of what a "successful response" looks like.

Where that gap actually comes from

It's not that these tools are badly built. It's that nobody designed for the seams. Auth works stage to stage differently. Errors mean different things depending on which vendor threw them. Retry logic that works for the parsing API silently breaks against the extraction API's rate limits. You end up writing the same glue code three times, and it's the least interesting code you'll write all quarter.

How we tried to close it with IDPForge

We built IDPForge around one rule: everything from parsing to post-processing sits behind the same API surface. One auth token. One response shape, consistently cased, across every stage. One error taxonomy, so a 422 means the same thing whether the document failed at extraction or at classification. Retry and idempotency behavior that doesn't change depending on which part of the pipeline you're calling.

That's not a small design choice. It's the difference between assembling a pipeline out of parts that were never meant to talk to each other, and calling one thing that already knows how its own stages fit together.

We didn't build this because we guessed developers would want it. We built it because we spent years being the ones stitching pipelines together, and we got tired of writing the same glue code every time.

Same Tuesday, same new document type. This time, lunch isn't spent debugging auth headers.

I’ve been working on a problem I kept seeing with tool-using agents:

An agent can understand the policy and still produce the wrong tool call.

If the action is consequential — refunding money, booking something, approving a request, modifying a record, calling a production API — observability after the fact is useful, but it’s already too late.

So I built ARK, an open-source runtime supervision layer that sits before execution.

The basic flow is:

agent proposes an action

→ ARK checks the applicable constraint + trusted evidence

→ ALLOW = execute

→ REJECT / REQUIRE_EVIDENCE = send feedback back to the agent

→ the agent decides again

One thing I intentionally avoided: ARK does not generate the replacement action.

The agent remains the author.

I tested this with LangGraph + an OpenAI model:

model proposed A

→ ARK rejected A before execution

→ feedback went back to the model

→ model authored B

→ ARK allowed B

→ only B executed

I’ve also been testing it on a scoped tau-bench airline failure class.

Paired K=16 result:

OFF: 1/16 passed (6.25%)

ON: 13/16 passed (81.25%)

9 directly attributable recoveries

0 observed regressions

I want to be careful with that result: it’s one constrained recovery failure class in a research benchmark, not a claim that ARK makes all agents reliable.

The SDK is public now:

pip install ark-agent-runtime

It currently works with custom Python agents and has a LangGraph integration.

I’m mainly curious how other people are handling this problem.

If you have an agent that can actually mutate production state, do you:

- validate tool arguments manually?

- use deterministic policy gates?

- rely on another model as a judge?

- sandbox actions?

- require human approval?

- just execute and monitor afterward?

I’d especially like feedback from people running agents that can refund, book, approve, purchase, or modify production data.

Site: arkruntime.com

GitHub: github.com/atripati/ark

Been building multi-agent RAG for a while and kept hitting the same wall: provenance tools tell you what happened, but nothing tells you how much to trust the result. A confident synthesis model at the end of a chain can’t repair a garbage extraction at the start of it — but nothing in the stack knows that.

Classical Islamic hadith science spent twelve centuries on a structurally identical problem: do you trust a statement transmitted through a chain of human narrators? Their answer was to grade every narrator individually, in a living registry, and cap the chain at its weakest link. No downstream reputation repairs an upstream liar.

So I built that as LangChain middleware. Every claim carries its chain (source → scraper → ingest model → answer model). Every transmitter has a per-domain grade that updates over time. The chain grade is the minimum across it, not the average. Fabricated chains get quarantined and the narrator gets flagged.

‘PiP install isnad’

It’s Apache-2.0, no API key, runs entirely local. Paper’s on arXiv (2607.24117) if you want the formal spec.

Happy to answer anything about the design — especially the parts I’m not sure about yet. Multi-provider narrator grading is still open.

A guardrail fires and blocks an output. The eval that flagged it lives in another tool, the trace in a third, so to see what happened you line all three up by hand. That is normal once an agent is in production: you run four things, tracing, evals, runtime guardrails, and a gateway in front of the models, usually four separate tools.

There's a real argument for keeping them separate. Each goes deeper in its own lane. Langfuse and Phoenix are strong at tracing, Ragas and DeepEval are real eval frameworks, Guardrails AI and NeMo handle policies, and Portkey and LiteLLM are solid gateways. Nothing locks you in, and you can swap any piece the week a better one ships.

The cost shows up later: four dashboards, four logins, data that never joins. Spend sits in the gateway, quality scores in the eval tool, the guardrail's decision elsewhere, nothing keyed the same way. The all-in-one bet is the opposite: the layers share context, so a trace, its eval score, and its guardrail decision sit in one record.

We build one of these, Future AGI: it runs tracing, evals, runtime guardrails, and the model and tool gateway in one Apache-2.0 stack you can self-host, so a blocked call never leaves and the trace lines up with the eval. It is still a nightly build with rough edges, and the honest reason to run it this way is fewer moving parts, not any single piece beating the dedicated tool.

So when a guardrail blocks something, is it on the same trace as the eval that flagged it, or are you piecing it together from separate tools? And if you consolidated, did it ever cost you on depth, where the bundled piece was weaker than what you gave up?

so i realized every "agentic payments" startup is just trying to sit in the middle and clip a few cents per transaction. the card tokenization part is already solved — Stripe does that. the real problem is nothing stops your agent from buying 100k of something or getting prompt-injected by a sketchy product page.

built a rules engine for it. agent wants to buy something, it checks your policy (spending limits, merchant restrictions, velocity controls, time windows) and returns ALLOW, DENY, or ESCALATE to a human.

Python, zero deps, MIT. pip install pyagentgate

https://github.com/Peterc3-dev/agentgate

felt like this should be open infrastructure before someone locks it down.

Check it

I have been trying a bunch of harness models and frameworks for the last month, and I realize most agent frameworks want you to rebuild your runtime around their harness.

We went the other way with AURA Harness: a thin membrane around loops you already run, plain Python, Ollama, LangGraph, whatever. You keep the body/runtime, while AURA records and optionally gates what crosses the boundary. Shouldn't this be what harness is about??

Local-first by default:

• Works offline with agent_ref + session IDs, no cloud identity required
• integrations/ollama/llama_loop.py, stdlib HTTP against Ollama (llama3.2:1b is our dev default)
• Optional verified operator identity if you need it later, not required for OSS/dev

What you get on close: JSONL spine, audit report, hash chain, aura verify chain for CI.

Loose coat = audit-only logging. Tight coat = rules/gates at egress when you wire tool paths.

Open source (Python): github.com/ARPAHLS/aura

Happy to answer setup questions, especially Ollama related + “wrap my script, don’t replace it.”

Contributors more than just welcome, bunch of good first issues open <3

The Problem: Keeping your Python functions and your OpenAI/Anthropic tool JSON schemas in sync is a nightmare. A missing required field or a typo in the schema breaks the LLM's ability to call your tool.

The Solution: I wrote a zero-dependency micro-tool that uses Python's built-in inspect module to read your functions and generate the exact JSON schema required by the APIs.

Features: * Generates OpenAI format (also works for Groq/Mistral/Ollama). * Generates Anthropic format (Claude 3.5 input_schema). * Reads type hints to map Python types to JSON Schema types. * Checks for default values: if a parameter has no default, it automatically adds it to the required array.

Just pass the function to the generator and hand the output directly to the API.

Repo: github.com/Encephos/function-schema-generator