LLM (semi) structured training and inference
Document AI / IE
Getting things out of documents
Key
- EVAPORATE: [2304.09433] Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes 1
-
We identify two fundamentally different strategies [..]: prompt the LLM to directly extract values from documents or prompt the LLM to synthesize code that performs the extraction.
- schema found automatically
-
Our key insight is to generate many candidate functions and ensemble their extractions using weak supervision. EVAPORATE-CODE+ not only outperforms the state-of-the art systems, but does so using a sublinear pass over the documents with the LLM.
-
To reduce variance, we synthesize many candidate functions, then estimate their quality and aggregate their extractions using weak supervision.
-
Benchmarks
Structured I/O
- [2505.20139] StructEval: Benchmarking LLMs’ Capabilities to Generate Structural Outputs 2 (10/10)
- generation (text-to-format) vs conversion tasks (format1-to-format2)
- A LOT of formats, text-based and visual-based ones
- text: JSON,CSV,TOML etc
- visual: SVG, latex, mergmaid, tikz, typst, vue, react etc.
- Evaluation
- for visual ones: VQA / ask a vision model whether the generated picture fullfills criterium.
- for text: dot-path
- Cool task generation prompt (“pick a super-creative and random domain”)
- Takeaways
- generation harder than conversion, vision harder than text
- [2411.19504] TQA-Bench: Evaluating LLMs for Multi-Table Question Answering3 (10/10)
- Tests multi-table reasoning with tables represented as Markdown, CSV, JSON, HTML
- Tasks like aggregation (count, sum avg), lookup, top-selection, correlation
- Results
- finds Markdown to be the best and JSON to be the worst for multi-table QA
- CSV/JSON’s relative ordering changes based on task type etc.
- Table-specific models underperform; distillation has a negative impact for long contexts
- Instruct+Code models the best, Chat-models consistently bad
- Multi-table reasoning is harder than single-table to the extent that merging 2 tables into 1 can improve scores
- finds Markdown to be the best and JSON to be the worst for multi-table QA
- Useful bits
- token budget defined by markdown representation and then converted to whatever (that will take more tokens)
- lists sources of relational databases, has cool algorithm to sample them
IE / Document extraction
-
[2604.25359v1] The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models 4 (8/10)
- Takeaway: often schema compliance but silently wrong values
- Neat overview of what’s wrong with other benchmarks
- Three modalities (image+audio)
- Thorough evaluation metrics, includes related work for e.g. constraint adherence or e.g. comparing JSONs
-
[2607.29677] ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction 5 , which is basically “given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata” and this is evaluated. Really cool, including classifying documents by how exactly are they problematic
- GitHub - run-llama/ExtractBench: ExtractBench - A Benchmark for Schema-Guided Enterprise Document Extraction · GitHub
- Demo: ExtractBench — Schema-Guided Extraction Benchmark for Enterprise Documents
Unstructured and semi-structured
Surveys
- [2508.00217v1] Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challenges 6(2025)
- exceptional
- Large Language Models for Structured and Semi-Structured Data, Recommender Systems and Knowledge Base Engineering: A Survey of Recent Techniques and Architectures7 Section 5
- Lists many open topics
- Large Language Model for Table Processing: A Survey (2024)8 nice
Processing
- TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios9 LLama 8B finetuned for querying+editing tables
- SpreadsheetLLM: Encoding Spreadsheets for Large Language Models10 is BRILLIANT
- Spreadsheets != tables
- Uses an inverted index (
bob: C1,C4,A0), homogeneous columns (e.g. all dates), automatically find the useful table bit in the sheet. Genuinely a lot of cool ideas. (Also SpreadsheetCompressor to save tokens)
Table formats/representations
- Blog posts by the same company:
- Which Table Format Do LLMs Understand Best? (Results for 11 Formats), includes list of formats and sample data
- doesn’t convince me
- only gpt4.1-nano
- ‘Markdown-KV’ won, XML was second, CSV was the second worst
- Which Nested Data Format Do LLMs Understand Best? JSON vs. YAML vs. XML vs. Markdown
- JSON, CSV, Markdown, YAML
- 3 models tested, results varied
- Which Table Format Do LLMs Understand Best? (Results for 11 Formats), includes list of formats and sample data
Etc.
- Text-to-table is an established task.
- [2508.15910v1] Evaluating Structured Decoding for Text-to-Table Generation: Evidence from Three Datasets 11
- discussion on limited suitability of usual NLP metrics on table level
- does neat postprocessing to find and check the markdown tables returned by the LLM
- [2508.15910v1] Evaluating Structured Decoding for Text-to-Table Generation: Evidence from Three Datasets 11
Etc.
Venues for relevant literature
- This footnote from Section 2.1 of 12:
Consider, for example, the International Conference on Document Analysis and Recognition (ICDAR), the International Journal on Document Analysis and Recognition (IJDAR), the International Workshop on Document Analyis Systems (DAS), the International Workshop on Graphics Recognition (GREC), some of which host serial competitions, such as the Robust Reading Competition linking the document analysis and computer vision communities [Ka13; Ya17], cf. https://rrc.cvc.uab.es/.
- Robust Reading Competition does really neat competitions
“Robust Reading” refers to the research area dealing with the interpretation of written communication in unconstrained settings. // e.g. scans maps etc. + VQA
- Start | The ACM Symposium on Document Engineering
-
Document engineering is the computer science discipline that investigates systems for documents in any form and in all media. As with the relationship between software engineering and software, document engineering is concerned with principles, tools and processes that improve our ability to create, manage, and maintain documents.
- Proceedings, e.g. 2025: Proceedings of the 2025 ACM Symposium on Document Engineering | ACM Conferences
- Sessions on IR, classification, analysis
-
Less relevant but interesting topics
Tabular foundation models
TL;DR: pretrained on large tabular datasets, predict via in-context learning (dynamically adapting to a task without optimizing any parameters). Do classification regression etc.
- Tabular Foundation Models the WIP book on the basics.
- Models
Long tail of weird stuff
- TIL about TOON which tries to use fewer tokens than JSON/TOML/… (but CSV wins for small datasets)
TODO
- [2603.23047] Parametric Knowledge and Retrieval Behavior in RAG Fine-Tuning for Electronic Design Automation13
- Fine-tune a 7B RAG that beats lager model
- long quotes
- Metrics for retrieval incl. using triplet-based (response, context, query)
- neat intro and related work
Angles
- Tables
- Tables: long form vs wide form transformations, interplacy with context size
- Corner cases
- very long column names or cells
- very sparse table
- bad extraction -> training on this bad data -> ??
-
<_(@aroralanguagemodels2025) “Language Models Enable Simple Systems for Generating Structured Views of Heterogeneous Data Lakes” (2025) / Simran Arora, Brandon Yang, Sabri Eyuboglu, Avanika Narayan, Andrew Hojel, Immanuel Trummer, Christopher Ré: http://arxiv.org/abs/2304.09433 /
10.48550/arXiv.2304.09433_> ↩︎ -
<_(@yangstructevalbenchmarking2026) “StructEval: Benchmarking LLMs’ Capabilities to Generate Structural Outputs” (2026) / Jialin Yang, Dongfu Jiang, Lipeng He, Sherman Siu, Yuxuan Zhang, Disen Liao, Zhuofeng Li, Huaye Zeng, Yiming Jia, Haozhe Wang, Benjamin Schneider, Chi Ruan, Wentao Ma, Zhiheng Lyu, Yifei Wang, Yi Lu, Quy Duc Do, Ziyan Jiang, Ping Nie, Wenhu Chen: http://arxiv.org/abs/2505.20139 /
10.48550/arXiv.2505.20139_> ↩︎ -
<_(@qiutqabenchevaluating2026) “TQA-Bench: Evaluating LLMs for Multi-Table Question Answering” (2026) / Zipeng Qiu, Chenyue Li, You Peng, Guangxin He, Binhang Yuan, Chen Wang: http://arxiv.org/abs/2411.19504 /
10.48550/arXiv.2411.19504_> ↩︎ -
<_(@singhstructuredoutput2026) “The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models” (2026) / Abhinav Kumar Singh, Harsha Vardhan Khurdula, Yoeven D. Khemlani, Vineet Agarwal: http://arxiv.org/abs/2604.25359 /
10.48550/arXiv.2604.25359_> ↩︎ -
<_(@zhangextractbenchbenchmark2026) “ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction” (2026) / Boyang Zhang, Adrian Lyjak, Eli Stewart, Zhaoqi Li, Simon Suo: http://arxiv.org/abs/2607.29677 /
10.48550/arXiv.2607.29677_> ↩︎ -
<_(@wutabulardata2025) “Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challenges” (2025) / Xiaofeng Wu, Alan Ritter, Wei Xu: https://arxiv.org/abs/2508.00217 /
10.48550/ARXIV.2508.00217_> ↩︎ -
<_(@smajiclargelanguage2025) “Large Language Models for Structured and Semi-Structured Data, Recommender Systems and Knowledge Base Engineering: A Survey of Recent Techniques and Architectures” (2025) / Alma Smajić, Ratomir Karlović, Mieta Bobanović Dasko, Ivan Lorencin: https://www.mdpi.com/2079-9292/14/15/3153 /
10.3390/electronics14153153_> ↩︎ -
<_(@lulargelanguage2024) “Large Language Model for Table Processing: A Survey” (2024) / Weizheng Lu, Jing Zhang, Ju Fan, Zihao Fu, Yueguo Chen, Xiaoyong Du: https://arxiv.org/abs/2402.05121 /
10.48550/ARXIV.2402.05121_> ↩︎ -
<_(@zhangtablellmenabling2024) “TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios” (2024) / Xiaokang Zhang, Sijia Luo, Bohan Zhang, Zeyao Ma, Jing Zhang, Yang Li, Guanlin Li, Zijun Yao, Kangli Xu, Jinchang Zhou, Daniel Zhang-Li, Jifan Yu, Shu Zhao, Juanzi Li, Jie Tang: https://arxiv.org/abs/2403.19318 /
10.48550/ARXIV.2403.19318_> ↩︎ -
<_(@dongspreadsheetllmencoding2024) “SpreadsheetLLM: Encoding Spreadsheets for Large Language Models” (2024) / Haoyu Dong, Jianbo Zhao, Yuzhang Tian, Junyu Xiong, Shiyu Xia, Mengyu Zhou, Yun Lin, José Cambronero, Yeye He, Shi Han, Dongmei Zhang: https://arxiv.org/abs/2407.09025 /
10.48550/ARXIV.2407.09025_> ↩︎ -
<_(@oestreichevaluatingstructured2025) “Evaluating Structured Decoding for Text-to-Table Generation: Evidence from Three Datasets” (2025) / Julian Oestreich, Lydia Müller: https://arxiv.org/abs/2508.15910 /
10.48550/ARXIV.2508.15910_> ↩︎ -
<_(@perakisclassifyingfigures2023) “Classifying figures and illustrations in electronics datasheets: A comparative evaluation of recent computer vision models on a custom collection of 4000 technical documents” (2023) / Lymperis Perakis, Julian Balling, Frank Binder, Gerhard Heyer, Franz Kreupl: https://dl.gi.de/handle/20.500.12116/43114 /
10.18420/INF2023_186_> ↩︎ -
<_(@oestreichparametricknowledge2026) “Parametric Knowledge and Retrieval Behavior in RAG Fine-Tuning for Electronic Design Automation” (2026) / Julian Oestreich, Maximilian Bley, Frank Binder, Lydia Müller, Maksym Sydorenko, André Alcalde: http://arxiv.org/abs/2603.23047 /
10.48550/arXiv.2603.23047_> ↩︎