context extraction from 2D PDF engineering drawings (P&IDs/Blueprints)? Looking for ideas on graph DBs & CV workarounds.

[u/prajwlshetru]

Hey everyone,

I am currently working on a pipeline to extract context from engineering drawings (like P&IDs and technical blueprints) stored in PDFs to feed into an LLM.

Multi-modal LLMs are great, but they aren't quite there yet for heavy-duty engineering files. Here are a few observations and hurdles I’ve hit so far:

• Multimodal LLMs lack precision: Standard vision models can read the overall drawing, but they miss critical component tags, small text, and fine details.
• OCR/Text extraction isn't enough: Pulling just the raw text from drawing tags helps with basic Quality Control (QC), but it completely misses the spatial classification and structural relationships.
• The Planned Workaround: I am planning to use classical Computer Vision (like Hough Transforms) for shape and icon detection. The goal is to extract the symbols and text, map their connections, and build a Graph Database to give the LLM structured context.

Has anyone successfully built a pipeline for this? I would love to hear your thoughts on a few things:

1. Are there specific CV models or open-source frameworks you recommend for engineering symbol detection over traditional Hough transforms?
2. How are you handling the layout/relationship parsing to build your graph edges?
3. Any recommendations for structuring the final graph data so an LLM can query it efficiently via GraphRAG?

If you are working on something similar or have dealt with CAD/PDF parsing for AI, I'd love to hear your suggestions or brainstorm solutions!

[u/justgord]

Do you have a pretty large dataset for this ?

also .. not really photogrammetry per se .. maybe crosspost to computervision or machinelearning subs ?

[u/prajwlshetru → u/justgord]

I am using opens source drawing files from the internet.

I am sorry reddit suggested me to post it here

[u/RajeevParmarAI]

I’d probably avoid treating this as an OCR-first problem. For P&IDs the difficult part is recovering topology, so I’d structure it as vector/CV extraction → symbol detection → connector/line extraction → spatial relationship engine → engineering graph → GraphRAG. If the PDF is vector-based, I’d extract primitives and text coordinates directly before rasterizing anything. For scanned drawings, use a detector for equipment/symbols and line extraction for pipes/connectors. Then associate OCR tags with detected objects using bounding boxes, proximity and domain rules. For the graph, I’d keep nodes such as Equipment, Instrument, Valve, Pipe and Junction, with edges like CONNECTED_TO, UPSTREAM_OF, MEASURES and CONTROLS. I’d avoid asking the LLM to infer raw topology; geometry/CV should establish that deterministically, while the LLM handles normalization, ambiguous cases and querying. One thing I’d be careful about is crossings vs actual junctions — that can create a lot of false graph edges. If the PDF retains vector information, that may be much easier to solve than from pixels alone.