r/Python • u/Sea-Possession-2536 • 7d ago
Discussion pdftotext -layout still isn't enough when a government PDF is secretly a 3-column table
Working through an old government safety document today and the raw text order was completely scrambled, hazard descriptions and countermeasures kept interleaving mid sentence. Turned out the whole page is a 3-column table, and if you don't slice by column x-position it just reads left to right across the whole line like it's one paragraph.
Fixed that with -layout and cutting at the column boundaries, but then hit a dumber problem: some lines have two □ markers on them, one for the subcategory heading and one for the first hazard item. Grab the first □ naively and the subcategory swallows the whole first hazard entry, and you don't notice until you're 40 rows in and something reads wrong.
Also found two pages where a single set of countermeasures gets shared by two different subcategories, which nothing in the text structure hints at, you just have to know the source document does that.
1
u/hakesson 5d ago
Different approach that worked for me on a tricky pdf. I asked an LLM to reproduce it as Typst and keep going until the generated images of the two documents were very close. The Typst source ends up being the extraction since the columns and reading order come out as real structure. It worked out nicely and was easy to verify but very token consuming (no surprise there), so maybe not good for running over a whole batch.