Web-scraped fine-tuning broke on boilerplate, Reddit ML post finds

A Reddit ML poster reports that fine-tuning on self-collected web data produced badly structured outputs because boilerplate text leaked into the training examples, and the problem only became clear after the model began generating output resembling half-rendered HTML. It matters because web scraping is often treated as a simple collection step, while this account suggests that what enters the dataset can shape the model's output format long before training starts.

What broke in the fine-tuning run?

According to the Reddit ML post, the initial assumption was that self-collected web data would be straightforward to use for fine-tuning. That assumption did not hold in practice. The poster describes a mismatch between the raw material collected from the web and the examples the model was effectively trained on.

The visible symptom was in generation. The model started producing weirdly structured outputs that looked like half-rendered HTML, according to the Reddit ML post. That symptom is useful because it points away from a narrow wording problem and toward the composition of the dataset itself. If training examples contain repeated page furniture, the model may learn to reproduce that furniture as part of the expected output.

For practitioners, the lesson is to inspect the training data as a product in its own right, rather than treating scraping as a neutral way to obtain text. A fine-tuning run can appear to fail at the model stage when the underlying examples already contain the wrong structure. Checking samples before training may therefore be more informative than changing model settings first.

Why did boilerplate leak into the training data?

The Reddit ML post identifies boilerplate contamination as the central problem. Navigation menus, cookie banners and footer text were leaking into training examples, according to the poster. The post calls this the silent killer, because such material can be present across many pages without being obvious in a quick glance at individual documents.

Boilerplate is repetitive by nature. Menus tend to repeat links and labels, banners tend to repeat notices, and footers tend to repeat site information. When that material is mixed with the content a researcher actually wants, the dataset gains a strong pattern that is unrelated to the intended task. A model exposed to many examples with that pattern may treat it as part of the normal shape of an answer.

This also explains why the resulting outputs looked like half-rendered HTML. The problem was not simply that the model used unfamiliar words. The structure itself had been contaminated. Text extracted without regard to page structure can preserve the scaffolding of a website while losing the distinction between content and surrounding interface elements.

A practical response is to ask, for each source, which parts of the page are meant to be learned from and which parts are merely the container. That question needs to be answered before extraction, because once boilerplate is flattened into plain text, it becomes harder to separate from the intended content.

What fixed the HTML structure problem?

According to the Reddit ML post, the poster solved the problem with aggressive DOM-targeting rather than raw text extraction. DOM-targeting means selecting the relevant parts of the document structure instead of accepting all visible text in the order a simple extractor returns it.

The distinction matters because a web page is not a single block of text. It contains headings, paragraphs, menus, sidebars, banners and footers, each in different parts of the document. Raw extraction can collapse those distinctions. Targeting the document structure gives the pipeline a chance to keep the intended content and exclude the surrounding elements that caused contamination.

The poster also changed how the pages were fetched. Switching to a rotating residential proxy layer made a noticeable difference in how clean the raw HTML was before parsing, according to Reddit ML. The post explains that per-request IP rotation means target sites serve the same rendered DOM a real browser session would get, rather than a bot-filtered skeleton.

That explanation separates two stages that are easy to conflate. The first stage is obtaining HTML that resembles what a real browser would receive. The second stage is parsing that HTML into examples. In this account, improving the first stage made the second stage easier because the raw HTML was cleaner before the parser touched it. It does not suggest that fetching alone solves boilerplate, since DOM-targeting was still needed. The combination addressed both the quality of the source document and the selection of content from it.

Why did deduplication order matter?

The Reddit ML post says deduplication order mattered more than the poster expected. It specifically mentions running exact dedup before fuzzy dedup, but the available summary ends at that point, so no further outcome should be assumed from the post.

Even without a stated result, the point is worth taking seriously for anyone building a dataset from the web. Exact duplicates and near-duplicates are different problems. Exact matching can remove identical copies, while fuzzy matching is intended to catch examples that are similar but not identical. The order in which those steps are applied can change which examples remain and how much near-repeated material survives into training.

For a web-scraped dataset, repetition is a particular concern because templates, menus and footers already create repeated text across pages. Deduplication therefore interacts with boilerplate removal. If repeated page furniture is not removed first, it may also distort duplicate detection. If duplicates are handled in an order that leaves many near-copies behind, the model may see the same pattern far more often than the underlying content warrants.

The cautious takeaway from the Reddit ML post is procedural rather than numerical. The poster found that order mattered, so teams should treat deduplication as a designed part of the pipeline and record the sequence they used. That makes later failures easier to diagnose, because the dataset can be compared against the intended cleaning steps instead of being treated as an opaque collection of pages.

How can you apply this in Grafy?

The most direct Grafy connection is through Grafy Code, which is described as an AI coding studio. A reader could use that workspace to prototype the kind of pipeline described in the Reddit ML post: fetching pages, targeting the relevant DOM structure, removing boilerplate, and applying exact and fuzzy deduplication in a recorded order. Grafy Code should not be assumed to provide the proxy layer or scraping behaviour described in the post, but it is a suitable place to write and test the parsing and cleaning logic that sits between collected HTML and training examples.

For the research and review side, Grafy Reader can be used to read, search and question PDFs and papers with AI, which may help a reader compare the anecdote with published data-cleaning practice before changing a pipeline. Grafy Chat can also be used to discuss the approach with many AI models side by side, for example by asking for a checklist of boilerplate types to look for in a sample of scraped pages.

That division of labour keeps the Grafy connection honest. The Reddit ML post is about data collection, parsing and deduplication, not about a Grafy feature that performs those steps automatically. Grafy is useful here as a place to build, inspect and discuss the workflow, while the technical decisions about fetching, DOM-targeting and deduplication remain the reader's responsibility.

Frequently asked questions

What caused the model to generate half-rendered HTML-like outputs?

According to the Reddit ML post, navigation menus, cookie banners and footer text leaked into training examples. That boilerplate contamination taught the model unwanted structure, so its outputs began to look like half-rendered HTML rather than clean text for the intended task. The poster described this as the silent killer in the collected dataset.

What did the poster change instead of raw text extraction?

The poster switched to aggressive DOM-targeting rather than raw text extraction, according to Reddit ML. Instead of taking page text as-is, the approach targeted the relevant parts of the document structure, reducing the chance that menus, banners and footers would enter examples. This was presented as the main parsing change alongside cleaner source HTML.

Did switching to a rotating residential proxy layer improve the raw HTML before parsing?

Yes. According to Reddit ML, switching to a rotating residential proxy layer made a noticeable difference in how clean the raw HTML was before parsing. The poster explained that per-request IP rotation means sites serve the same rendered DOM a real browser session would get, rather than a bot-filtered skeleton.

---

Source: Reddit ML — What actually broke when I tried to fine-tune on web-scraped data and what fixed it [D]