Data Handling
Text Lab promises its users that files they upload are processed in their own session and not left behind. This page describes how the code keeps that promise and what a developer must do to keep it true.
Where data may go
| Data | Location | Removed |
|---|---|---|
| Uploads, intermediate files, generated charts | The job workspace (textlab.common.storage) |
By the feature when a run ends, and always when the session ends |
| Results shown to the user | Streamlit session state | When the page is reloaded or closed, or the session ends |
| Output the user deliberately saves | A location the user chooses (Knowledge Graph output folder) | When the user deletes it |
| Logs | The session's job folder in the user's home directory | When the user deletes it |
| Model files the user downloads (custom Topic Modeling embeddings) | The user's Hugging Face cache | When the user deletes them |
Nothing else may be written: not to the home directory, not to the source tree, and not to shared folders on research storage.
The job workspace
At the start of a session the launch script creates one private folder for
the job and exports its path as TEXT_LAB_WORKDIR:
- It is created in the job's
$TMPDIR, or inTEXT_LAB_WORKDIR_BASEwhen the site configuration sets it. - It has owner-only permissions (
700). - A
trapin the launch script deletes it when the session ends, including when the job is cancelled or reaches its time limit. - On UBELIX, Slurm's epilog also deletes
$TMPDIR(/scratch/local/<job>) after every job, so even a killed job leaves nothing behind.
The launch script also points TMPDIR inside the container at
$TEXT_LAB_WORKDIR/tmp. Code that uses Python's tempfile module with its
defaults, Ollama and Streamlit's own temporary files therefore all land in
the workspace.
Using it in code
from textlab.common.storage import get_workspace
workspace = get_workspace()
# A folder for a feature area, created on first use (owner-only):
jobs_dir = workspace.dir("ocr")
# A folder removed when the block ends, also on errors:
with workspace.temp_dir("visualization") as run_dir:
...
# A uniquely named folder that lives for a whole session, for example a
# chat session's uploads; removed with the workspace at the latest:
session_dir = workspace.make_temp_dir("chat", prefix="chat-")
Each feature uses its own area name, so features cannot delete each other's
files. Worker processes (textlab.common.jobs) exchange their request,
progress and result through a job folder in the caller's area, removed when
the run ends; the transcription feature stages uploads the same way
(staged_uploads, staged_zip), translation writes the PDF pages it
sends to OCR to translation/ocr-*, OCR keeps each run in ocr/job-*,
ocr/batch-* or, for manual engine selection, ocr/manual-*, Survey
writes the blank form's images and rebuilt exports to survey/*, and Topic
Modeling writes the table it hands to its worker, and the result ZIP, to
topic_modeling/run-*. Visualize Data keeps each run's table and charts in
visualization/run-*, and Chat a conversation's table and charts in
chat/conversation-*, removed when the user starts a new chat.
Temporary folders are removed even when a tool left read-only files in them
(storage.remove_tree). Without TEXT_LAB_WORKDIR (tests, scripts run by
hand), the workspace is a private folder under the system temporary
directory.
A workspace folder that already exists but belongs to another user is
refused with WorkspaceError, so a shared /tmp cannot be used to read
another user's files.
What the tests enforce
tests/test_data_footprint.py scans the source code and fails if:
- code builds a path from the home directory (
Path.home(),expanduser("~"),$HOME,$XDG_CACHE_HOME), or - a string names the UBELIX site (
/storage/,unibe.ch,ubelix) instead of reading it from the site configuration.
The few legitimate uses are listed in the test with their reason, for example the Topic Modeling cache for model files. An allow-list entry that no longer matches anything also fails the test, so the lists stay accurate.
src/textlab/common/tests/test_storage.py checks the workspace itself:
permissions, cleanup of temporary folders on errors, and the refusal of
folders owned by someone else.
Code written by a model
Text a model reads can steer it (prompt injection): a cell of an uploaded
table can tell it to write harmful code. Model-written code is therefore
never run in the app's own process. The Visualization feature's custom
chart tools check the code first and run it in a sandbox: a separate
container started with apptainer exec --containall --no-home --net
--network none with a clean environment (inside a container, Apptainer's
APPTAINER_BIND would pass on the outer container's bind mounts), which
sees a copy of the table and nothing else of the
user's files, and has no network (see features/visualization/sandbox.py).
Any new feature that runs model-written code must use the same sandbox.
When you add or change a feature
- Write temporary files only through
get_workspace(), in the feature's own area. Prefertemp_dir()so files disappear as soon as they are not needed. - Do not log file contents or model output wholesale; logs stay in the user's home directory after the session.
- If the feature sends data anywhere outside the session (like the Knowledge Graph's GPUStack option), say so in the feature's user guide page and in the README.
- Update the privacy section of the feature's user guide page and the table in the README when what is stored, or where, changes.