Skip to content

Testing

Where tests live

  • Feature tests live next to their feature, in src/textlab/features/<feature>/tests/.
  • Cross-cutting tests live in tests/: checks that no user data is written outside the job workspace and that no site-specific values are hardcoded (test_data_footprint.py), and that the site configuration provides every setting (test_site_config.py).

Shared code has its tests in src/textlab/common/tests/. Each tests/ folder is a package (it has an __init__.py), so a test can import helpers from another test module by its full name, for example from textlab.features.translation.tests.test_translation_pdf import make_pdf.

Markers

Tests that need more than plain Python are marked, so they can be selected or skipped:

Marker Meaning
gpu Needs a CUDA GPU
container Needs the Text Lab image (models, binaries such as Tesseract)
ollama Needs a running Ollama server
slow Takes more than about a minute

Unregistered markers are an error (--strict-markers), so add new ones to pyproject.toml first.

Most tests replace models and servers with fakes, so they run anywhere: for example features/transcription/tests/test_service.py fakes WhisperX and ffmpeg, and common/tests/test_ollama.py fakes the Ollama client. A few tests run the real thing and carry the container marker, such as features/transcription/tests/test_integration.py; they skip themselves when the model stores are not mounted. Tests of worker processes start real subprocesses with a stub worker (common/tests/stub_worker.py). The PaddleOCR-VL and PaddleOCR workers are tested the same way: features/ocr/tests/ writes stub scripts that speak their protocol and passes them as worker_path.

Running the tests

The tests run inside the Apptainer image on a compute node, never on a login node. scripts/test_on_node.sbatch wraps the command.

One-time setup

The image does not include pytest. Install it once into your home directory, from a compute node:

SIF=/storage/research/dsl_shared/solutions/ondemand/text_lab/container/text_lab_210526.sif
apptainer exec --env PYTHONNOUSERSITE=1 "$SIF" \
  /opt/conda/envs/text_lab_main/bin/python -m pip install \
  --target "$HOME/.cache/text_lab_pytest" pytest

Warnings from pip's dependency resolver during this install are harmless.

As a batch job

From the repository root:

sbatch --qos=job_gratis scripts/test_on_node.sbatch
sbatch --qos=job_gratis scripts/test_on_node.sbatch -m "not slow"

The output goes to textlab-tests-<jobid>.out in the repository root. Arguments after the script name are passed to pytest.

In an interactive session

From an srun or salloc session on a compute node:

bash scripts/test_on_node.sbatch -k translation

The script refuses to run outside a Slurm allocation. It reads the site configuration (deploy/site.env in the repository, or TEXT_LAB_SITE_ENV) for the image and the model stores, and mounts them as the app does, so the container tests find the models. TL_CONTAINER selects a different image, for example a newly built one.

Lint and architecture checks

These are lightweight and also run on GitHub for every push:

ruff check .            # PEP 8, docstrings, import order, likely bugs
ruff format --check .   # formatting
lint-imports            # backend never imports a user interface

lint-imports needs src on the Python path (PYTHONPATH=src). Code not refactored yet is excluded from ruff (extend-exclude in pyproject.toml); refactoring a module removes it from that list.

Continuous integration

.github/workflows/ci.yml runs on every push and pull request:

  • ruff lint and format checks
  • a syntax check of all Python files
  • the import-linter contracts
  • a strict build of this documentation site

GitHub's machines have no GPU and no image, so the tests do not run there: run them on a compute node and give the Slurm job ID in the pull request.

Pull requests

main is protected: changes arrive through pull requests, and a pull request can be merged only when both CI jobs pass. The pull request template (.github/pull_request_template.md) asks for what CI cannot check: the test job ID, a UI check in a sandbox app, updated documentation, no emojis, and an explanation of any change to model-facing prompts.