Built by Maven

Content

Overview

Extracting data from web pages is one of the primary use cases for HtmlUnit. Because HtmlUnit models a real web browser (including DOM tree generation, CSS evaluation, and JavaScript execution), it offers powerful capabilities for extracting structured data, raw text, HTML fragments, and element attributes.

Extracting Rendered Text with asNormalizedText()

The primary method for extracting human-readable content from any DOM node or page is asNormalizedText().

Unlike standard raw text getters, asNormalizedText() simulates how text is actually rendered on screen:

  • Collapses consecutive whitespace sequences into single spaces.
  • Appends line breaks (\n) between block-level CSS elements (such as <p>, <div>, or <tr>).
  • Ignores invisible DOM elements like <script> and <style> tags.
  • Respects CSS visibility settings, excluding content hidden via rules such as display: none or visibility: hidden.

Extracting Text from a Targeted Element

For targeted extraction, combine CSS or XPath selectors with asNormalizedText(): 

try (final WebClient webClient = new WebClient()) {
    final HtmlPage page = webClient.getPage("https://www.htmlunit.org/");

    // Locate element via CSS Selector
    final DomNode article = page.querySelector("article.main-content");
    if (article != null) {
        String text = article.asNormalizedText();
        System.out.println(text);
    }
}

Extracting Text from the Entire Page

Calling asNormalizedText() directly on the HtmlPage or HtmlBody retrieves all visible text across the document. 

final String fullPageText = page.asNormalizedText();

Extracting HTML / XML Source

If you need the underlying markup rather than plain text, HtmlUnit provides asXml() on all DOM nodes.

Serializing the DOM with asXml()

Calling asXml() serializes the current state of a node (and its child subtree) into valid XML/HTML markup. Because it reads from the active DOM tree, it includes any dynamic changes made by JavaScript after page load.

  • Calling page.asXml() returns the entire rendered document markup.
  • Calling element.asXml() returns the HTML/XML representation of that specific element and its descendants. 
// Get full serialized XML/HTML representation of the page DOM
final String pageXml = page.asXml();

// Get the HTML/XML code of a specific element and its children
final HtmlDivision div = page.getHtmlElementById("content");
final String divXml = div.asXml();

Extracting Attributes and Links

When scraping web sites, extracting element attributes (such as href, src, or data-* attributes) is common. 

// Extract all links with their target URLs
final List<HtmlAnchor> anchors = page.getAnchors();
for (final HtmlAnchor anchor : anchors) {
    String text = anchor.asNormalizedText();
    String href = anchor.getHrefAttribute();
    System.out.println(text + " -> " + href);
}

// Extract specific image sources
final DomNodeList<DomElement> images = page.getElementsByTagName("img");
for (final DomElement img : images) {
    String src = img.getAttribute("src");
    String alt = img.getAttribute("alt");
}

Navigating Links: Prefer .click() over Extracting href

With traditional raw scrapers, navigating to a new page typically requires extracting the href attribute string and manually issuing a new GET request.

Because HtmlUnit simulates a real browser, in most cases you do not need to extract the href attribute. Instead, simply call click() directly on the anchor element: 

// Find an anchor by its text or selector
final HtmlAnchor link = page.getAnchorByText("Documentation");

// Simulate a real user click - handles JavaScript onclick handlers, relative URLs, and redirects automatically
final HtmlPage targetPage = link.click();

Using click() is strongly recommended over extracting `href` manually because it:

  • Triggers any attached JavaScript onclick event handlers.
  • Handles single-page application (SPA) client-side routing.
  • Resolves relative paths, base tags, and redirects automatically.
  • Maintains browser session state, cookies, and window history.

Structured Extraction with Tables

For tabular data, HtmlUnit provides specialized wrappers like HtmlTable, HtmlTableRow, and HtmlTableCell that simplify extracting rows and columns. 

final HtmlTable table = page.getHtmlElementById("data-table");
for (final HtmlTableRow row : table.getRows()) {
    for (final HtmlTableCell cell : row.getCells()) {
        System.out.print(cell.asNormalizedText() + "\t");
    }
    System.out.println();
}

For deeper table manipulation details, see the dedicated Tables guide.

Combining HtmlUnit with Jsoup

If you prefer Jsoup's fluent CSS selector and DOM manipulation API for content extraction, you can use the official htmlunit-jsoup bridge. This allows HtmlUnit to handle JavaScript execution and browser navigation, while providing a Jsoup Document interface for extraction.

Where to Go Next

  • Getting Started - Learn the basics of loading pages and interacting with forms.
  • Use JavaScript - Ensure dynamic content is fully loaded before extracting text.
  • Use Tables - Advanced table extraction techniques.