Setup and usage

Here is an example test for matplotlib graphics, from the tinyvdiff demo project:

import pytest

from tinyvdiff_demo.matplotlib import generate_plot


@pytest.fixture
def temp_pdf(tmp_path):
    """Fixture to create a temporary PDF file path"""
    return tmp_path / "test.pdf"


def test_matplotlib_visual(tinyvdiff, temp_pdf):
    """Test visual regression with PDF generated with matplotlib"""
    pdf_path = generate_plot(temp_pdf)
    tinyvdiff.assert_pdf_snapshot(pdf_path, "snapshot_matplotlib.svg")

See the setup guide to learn how to integrate tinyvdiff into your Python project and use it in CI/CD systems like GitHub Actions.

Motivation

While transitioning parts of my toolchain to Python, I encountered a need for a unified approach to test the visual appearance of rendered outputs, such as graphics and documents. In R, tools like testthat snapshot tests and vdiffr address similar needs effectively. However, with all the flexibility, I saw an opportunity to explore this space a bit more in Python.

The trilemma

The design considerations for tinyvdiff are detailed in its design document. In brief, I identified three key requirements that define a trilemma for visual regression testing:

1. Support for diverse input formats: Handle graphics and documents produced by different backends.
2. Bitwise reproducibility of snapshots: Store snapshots in plain text for easy version control and comparison.
3. Environment consistency: Generate consistent snapshots across operating systems.

My solution

Given the constraints of this trilemma:

• Dropping constraints 2 and 3 leads to testthat “whole file snapshotting”.
• Dropping constraint 1 while keeping 2 and 3 leads to vdiffr.
• Dropping constraint 3 while keeping 1 and 2 gives tinyvdiff.

tinyvdiff operates on the assumption that visual regression tests will be running on a single platform. Developers are also trusted to prepare reproducible PDF inputs in their preferred way. The package then uses pdf2svg (built on Poppler and Cairo) to convert PDF files into SVG snapshots for comparison.

Use cases

tinyvdiff may be a good fit for projects where:

• Visual tests can be sufficiently done on a single platform.
• Multiple types of visual content require appearance testing.
• PDF inputs are reproducible and contain mostly vector elements.
• Snapshots remain manageable in size and complexity.

If any of these conditions are not met, other more focused approaches might better suit your needs. For example, matplotlib has built-in testing helpers for image comparison, which works great for testing the appearance of its outputs.