PDF to JSONL for RAG and fine-tuning

JSONL is the usual way to hand a set of passages to a search index or a training job. This guide explains the format, the fields a RAG corpus needs, and two ways to get from a PDF to JSONL: a short Python script, or Agnos with no code. The examples are real output, and you can download them.

What JSONL is

JSONL (JSON Lines) is a text file with one JSON object on each line. There are no commas between the lines and no brackets around the file. Two lines from a file Agnos made from its sample project:

{"text": "## Orders\n\nQ: Is it refundable?\nA: Yes.\n\nQ: How long does a quote hold?\nA: Thirty days from the date on the quote [1]."}
{"text": "#### Clearance stock\n\nClearance lines are sold at the price shown and carry no further discount [2]."}

A few rules make a file valid:

• Each line is one complete JSON object. A line break inside the text is written as \n, so a record never spans two lines.
• The file is UTF-8 text.
• The name ends in .jsonl, sometimes .ndjson.

That shape is why datasets use it. A program can read one record at a time without loading the whole file, you can add records by appending lines, and one damaged line does not make the rest unreadable.

The fields a RAG corpus needs

A search index needs more than the text. Each record should say what it is and where it came from, so an answer can cite its source and you can update one document without rebuilding everything.

Most vector databases and RAG frameworks take a text plus a set of metadata fields, so these map across directly. Every field Agnos can write is listed in Export formats.

What Agnos writes for a PDF

This is the record Agnos wrote for the four-page handbook PDF from the headers and footers guide, cleaned with the RAG / Retrieval Corpus purpose. It is laid out across several lines here, with the text shortened; in the file it is one line.

{"text": "Northwind Tools Staff Handbook\nThis handbook covers the everyday rules … nobody keeps a copy after\nleaving.",
 "chunk_id": "d733867d65aa37daf2259108e3323ff6_000000",
 "source": "northwind-staff-handbook.pdf",
 "file_id": 1,
 "uploaded_at": "2026-09-23T23:02:03.677203Z",
 "quality_score": 0.8541,
 "language_confidence": 0.0,
 "content_type": "prose",
 "chunk_index": 0,
 "page_start": 1,
 "page_end": 4}

The headers, footers and page numbers are gone from text, but the pages are kept as fields: this piece runs from page 1 to page 4. The handbook is short, so it is one piece. A longer PDF is cut into several, each with its own page_start and page_end. language_confidence is 0 because language detection did not run.

A Word document has no fixed pages, so its pieces carry the headings they sit under instead. A record from the sample project’s staff-handbook.docx, shortened the same way:

{"text": "## 2. Expenses\n\n### 2.1 What we reimburse\n\nTravel booked through the workshop account.\n\n…",
 "chunk_id": "c8cb00647e803c487fad217ef8c7ff6d_000002",
 "source": "staff-handbook.docx",
 "file_id": 6,
 "uploaded_at": "2026-09-23T23:03:41.973756Z",
 "quality_score": 0.9258,
 "language_confidence": 0.0,
 "content_type": "prose",
 "chunk_index": 2,
 "heading_path": ["Northwind Tools Staff Handbook", "2. Expenses", "2.1 What we reimburse"]}

To see a whole file, download the sample PDF (7 KB) and the JSONL Agnos made from it (1.5 KB).

JSONL for fine-tuning

Training data is usually simpler: the text and nothing else. Agnos’s Training data export writes records like the two at the top of this page, {"text": ...}, one per line.

To test a trained model you need text it has not seen, so you can hold some files back as a validation set. Agnos splits by file, not by piece, and keeps files that share a passage on the same side. With the sample project’s five files, the handbook PDF and a split of 0.3, the command line reported:

  split                       4 train / 2 validation

The zip then holds train.jsonl, val.jsonl, and a manifest that counts any identical or nearly identical text found on both sides. Two things to know:

• Keep one copy of each document. The same handbook as a PDF and as a Word file is cut into different pieces, so it is not matched as a repeat. In this run the two copies landed on opposite sides of the split.
• Check the format your trainer expects. Many fine-tuning services for chat models want each line to hold a conversation, a list of messages, rather than plain text. Agnos writes plain text records. It does not write questions and answers or conversations.

Make it yourself with Python

This writes one record per page. It uses pdfplumber (pip install pdfplumber).

import json
from pathlib import Path

import pdfplumber  # pip install pdfplumber

source = Path("handbook.pdf")

with pdfplumber.open(source) as pdf, open("handbook.jsonl", "w", encoding="utf-8") as out:
    for number, page in enumerate(pdf.pages, start=1):
        text = (page.extract_text() or "").strip()
        if not text:
            continue  # a scanned page has no text layer
        record = {"text": text, "source": source.name,
                  "page_start": number, "page_end": number}
        out.write(json.dumps(record, ensure_ascii=False) + "\n")

Its first record for the handbook, with the text shortened:

{"text": "Northwind Tools | Staff Handbook | March 2026\nNorthwind Tools Staff Handbook\nThis handbook covers … the same week it is worked.\nConfidential - Internal Use Only\nPage 1 of 4", "source": "handbook.pdf", "page_start": 1, "page_end": 1}

It works, and it shows what is left to do:

• The header, footer and page number are in every record. Run the filter from the headers and footers guide first.
• A page is an arbitrary unit. A section that crosses a page break is split in two, and a page with three topics is one record. Cutting at headings, with a size limit, usually gives better passages.
• Tables come out as words separated by spaces. pdfplumber’s extract_tables() reads them as rows, but you decide how to write them.
• Passages repeated across files stay in, and a scanned page gives no text at all: it needs text recognition (OCR) first, which this script does not do.
• There is no id for each record. Add one if you will update the index later.

With Agnos, without code

1. Start a project. In Projects, press New project, type a name, check that the purpose is RAG / Retrieval Corpus (press Change if it is not), and press Create project.

2. Add your PDFs. Drop the files, or a folder of them, on the Import step.

3. Check what was removed. Open Review, choose a file and press Removed.

4. Export. Open Export, keep RAG corpus for a search index, or choose Training data and set a validation split for fine-tuning. Press Build and download.

The file is saved to your downloads with a name like agnos-dataset_20260923_141502.jsonl. Compared with the script, Agnos removes and lists the page furniture, keeps each piece’s pages, id and position, leaves out passages repeated across files, and reads Word, PowerPoint, Excel, web pages and text files as well as PDFs.

If your PDFs hold tables, set PDF reading mode to Layout in the project’s settings, under Reading files: it is slower, and it is the only mode that reads tables. Agnos reads text-based PDFs only: a scan has no text layer, and Agnos does not read text out of pictures. The command line runs the same steps from a script.