Deployment
Text Lab is an Open OnDemand interactive app. A session is a Slurm job that
runs template/script.sh.erb, which starts Ollama and the Streamlit app
inside an Apptainer image.
What lives where
| Part | Where | Who changes it |
|---|---|---|
Open OnDemand files (manifest.yml, form.yml, submit.yml.erb, view.html.erb, icon.png, template/) |
Deployed by the HPC team from the repository | Needs a redeployment by the HPC team |
Site configuration (site.env) |
Reference copy in deploy/site.env; live copy on shared storage |
The maintainers, at any time |
Application code (src/) |
A release folder on shared storage | The maintainers, at any time |
Apptainer image (.sif) |
Built from deploy/container/text_lab.def, stored on shared storage |
The maintainers, at any time |
| Models (Ollama, Whisper, Hugging Face, OCR) | Model stores on shared storage | The maintainers, at any time |
The deployed launch script contains a single site-specific value: the path of
the live site configuration (TEXT_LAB_SITE_ENV). Everything else is read
from that file, so changing code, the image, models or settings never needs
the HPC team.
On UBELIX, the shared folder is
/storage/research/dsl_shared/solutions/ondemand/text_lab
(TEXT_LAB_BASE):
text_lab/
├── site.env # live site configuration
├── current -> releases/vX.Y.Z
├── releases/ # one checkout per released version
├── container/ # images (.sif) and the model stores (models/)
└── logs/ # session logs of versions before 3.0.0
How a session starts
- The user fills in the launch form (
form.yml) and Open OnDemand submits a Slurm job (submit.yml.erb). - On the compute node, Open OnDemand copies
template/into the session's job folder in the user's home directory and runsbefore.sh.erb(port and session token) and thenscript.sh.erb. - The launch script loads
dev.envif one is next to it (sandbox apps only), then the site configuration, and checks that the release, the image and the model stores it names exist. - It resolves
currentto the release folder once, so the session keeps the version it started with even ifcurrentchanges meanwhile. - It creates the private job workspace, with a trap that deletes it when the job ends, and picks random ports for Ollama and Grobid.
- It starts the image with the model stores and the workspace bound, runs
ollama serveand, once Ollama answers, the Streamlit app (textlab/ui/streamlit/Home.py) from the release folder. after.shwaits for the app's port; the session card then shows Connect. The app checks the session token on every page (auth.py).
The top of output.log in the job folder lists the site configuration,
source tree, image, workspace and ports the session used.
What needs whom
| You want to | You change | HPC team needed? |
|---|---|---|
| Release new application code | A new release folder, then current |
No |
| Roll back to an earlier version | current |
No |
| Use a new image | Build it, copy it to container/, set TL_CONTAINER in site.env |
No |
| Add or update a model | The model store (see Model stores) | No |
| Change a setting (paths, endpoints, approved models) | site.env |
No |
| Change the launch form (GPU types, partitions, QoS, time limits) | form.yml, submit.yml.erb |
Yes |
| Change how a session starts (binds, ports, Ollama options) | template/ |
Yes |
| Move the site configuration | TEXT_LAB_SITE_ENV in template/script.sh.erb |
Yes |
Changes the HPC team deploys should be rare; prefer a setting in
site.env that the launch script reads.
The site configuration
deploy/site.env is a shell file sourced by the launch script. It sets:
- Locations used by the launch script: the base folder on shared storage,
the release to serve (
TEXT_LAB_SRC), the image (TL_CONTAINER), the model stores, extra paths to bind into the image, the portal host name, and optionally where the job workspace goes (TEXT_LAB_WORKDIR_BASE). - App settings, exported into the container and read by
textlab.common.config: the Hugging Face token file, the Swiss German Whisper models, the Grobid image, the GPUStack endpoint and the Ollama models used for OCR enrichment and surveys.
Every assignment has the form NAME="${NAME:-default}", so a value set
earlier (by template/dev.env) wins. tests/test_site_config.py checks
this, and that every setting the app reads is exported by the file.
Changing a setting
- Edit
deploy/site.envin your working tree. - Test it in a sandbox app (see below), which uses the repository copy.
- Commit the change.
- Copy the file to its live location,
$TEXT_LAB_BASE/site.env. New sessions use it; running sessions are not affected.
Adding a setting
- Add a
SettingSpecand a field totextlab.common.config. - Export it in
deploy/site.envwith a comment saying what it is for. - Read it in the code with
get_settings(), orget_settings().require()where the feature cannot work without it.
Deploying on another cluster
- Build the image from
deploy/container/text_lab.def(or copy it) and get a Grobid image if the Knowledge Graph is wanted. - Create the shared folder with
container/,releases/and the model stores, and fill the stores (see Model stores). - Copy
deploy/site.env, change the locations and settings, and put it where the deployed launch script expects it (or changeTEXT_LAB_SITE_ENVat the top oftemplate/script.sh.erbbefore handing the app to the HPC team). - Adjust
form.yml(cluster name, partitions, QoS and GPU types) for the cluster. If$TMPDIRis not set per job, or is too small, setTEXT_LAB_WORKDIR_BASE. - Make a release (see Releases) and ask the HPC team to deploy the Open OnDemand files.
Running your working tree in a sandbox app
template/dev.env (gitignored) overrides values before the site
configuration is read. Copy template/dev.env.example to template/dev.env
in your sandbox checkout and set what you want to test:
| To test | Set in dev.env |
Leave unset |
|---|---|---|
| Your code with the live configuration | TEXT_LAB_SRC (your src/) |
TEXT_LAB_SITE_ENV |
Your code and a changed site.env |
TEXT_LAB_SRC, TEXT_LAB_SITE_ENV (the repository's deploy/site.env) |
|
| A new image with the released code | TL_CONTAINER |
TEXT_LAB_SRC |
| A new image with your code | TEXT_LAB_SRC, TL_CONTAINER |
|
| Exactly what production runs (before a release goes live) | Nothing: remove dev.env |
Everything |
Open OnDemand copies template/ into the session's job folder, and the
launch script loads dev.env from there. Check the top of output.log to
confirm which source tree, configuration and image the session used.
The sandbox runs whatever is checked out in the working tree, so stay on the branch you are testing.
Model stores
Models are not part of the image: they live in stores on shared storage,
bound into the image at start (HOST_*_DIR in site.env). Some features
read files from outside TEXT_LAB_BASE, also set in site.env.
| Store | Holds | Used by | Where the code names the models |
|---|---|---|---|
models/ollama |
Ollama models | Chat, Visualize Data, Meeting Notes, Translation (LLM), OCR enrichments and GLM-OCR, surveys, Knowledge Graph (local) | common/models.json; TEXTLAB_VISION_MODEL, TEXTLAB_APPROVED_SURVEY_MODELS, TEXTLAB_GLM_OCR_MODEL in site.env |
models/whisper |
Whisper checkpoints (large-v3-turbo.pt, ...) |
Transcription, Meeting Notes | transcription/whisper_models.py |
models/whisperx, models/torch |
WhisperX and PyTorch caches | Transcription | WhisperX defaults |
models/huggingface |
Hugging Face cache: alignment models, diarization, translation models, olmOCR, Topic Modeling embeddings | Transcription, Translation, OCR (olmOCR), Topic Modeling | transcription/service.py, translation/engine.py, topic_modeling/models.py |
models/easyocr |
EasyOCR detection and recognition models | OCR (EasyOCR) | ocr/engines/easy_ocr.py |
models/paddleocr, models/paddlex |
PaddleOCR and PaddleX models, including PaddleOCR-VL | OCR (automatic pipeline, PaddleOCR) | ocr/ |
TEXT_LAB_CUSTOM_WHISPER_DIR |
Swiss German Whisper models | Transcription (Swiss German) | transcription/whisper_models.py |
TEXT_LAB_HF_TOKEN_FILE |
Hugging Face token for the gated diarization model | Transcription (speakers) | site.env |
Who can write
The launch script binds the stores read-write, so a library downloads a missing model into its store, but only for a user who may write there; for everyone else the download fails and the feature reports an error. Normal users can only read the stores. On UBELIX (checked in October 2026):
ollamaandwhisperare writable by the maintainers' group (rs_dsl_shared);- the other stores are writable only by the user who created them.
The stores' default ACLs give new files no permissions for other users, so a model a maintainer downloads is unreadable for everyone else until its permissions are opened (step 3 below).
Adding or updating a model
- Add the model to the code or configuration that offers it (for Ollama,
common/models.jsonor a setting insite.env). - As a user who can write to the store, download it:
- Ollama: start a Text Lab session and choose the model in Chat, which
pulls it; or run
ollama pull <model>inside the image withOLLAMA_MODELSpointing to the store. - Hugging Face, Whisper, EasyOCR, PaddleOCR: use the feature once in a session with the model selected; the library downloads it into the bound store.
- Ollama: start a Text Lab session and choose the model in Chat, which
pulls it; or run
- Make the new files readable by everyone, for example
chmod -R a+rX "$TEXT_LAB_BASE/container/models/<store>". - Release the code change as usual.
A fresh installation fills its stores the same way, one feature at a time.
A single command that downloads everything the code names is planned (see
cli.py); until it exists, the table above is the list to work through.
Logs
Each session writes streamlit.log and ollama.log next to Open OnDemand's
output.log, in the session's job folder in the user's home directory. Users
find the folder through the Session ID link on their session card and
send the logs with a problem report (see the user guide,
Reporting a Problem).
Releases
The site configuration serves $TEXT_LAB_BASE/current/src. current is a
symbolic link to a release folder, a checkout of a tagged version, so that a
release or a rollback is a change of the link. Sessions resolve the path at
start, so running sessions keep the version they started with.
Making a release
- Set the new version in
pyproject.toml(version = "X.Y.Z") in the last pull request of the release. - Merge the changes into
mainand check that CI passes. Merging publishes the documentation site, including the user guide. - Run the tests on a compute node (
scripts/test_on_node.sbatch). - Tag the commit (
vX.Y.Z) and create a GitHub release with notes. -
Check out the tag into a new release folder, readable by everyone:
cd "$TEXT_LAB_BASE/releases" git clone --depth 1 --branch vX.Y.Z \ https://github.com/dsl-unibe-ch/text_lab.git vX.Y.Z chmod -R a+rX,go-w vX.Y.Z -
If
deploy/site.envchanged, copy it to$TEXT_LAB_BASE/site.env. - Try the release in a sandbox app with
TEXT_LAB_SRCset to the new folder'ssrc/. -
Switch new sessions to it:
ln -sfn releases/vX.Y.Z "$TEXT_LAB_BASE/current" -
If the Open OnDemand files changed, ask the HPC team to deploy them from the tag.
Rolling back
Point current back at the previous release folder
(ln -sfn releases/vX.Y.W "$TEXT_LAB_BASE/current"). New sessions use it
at once; running sessions keep their version. Keep at least the previous
release folder until the new one has run without problems. If the release
also changed site.env, restore the previous copy too.
If a release needed the HPC team (changed Open OnDemand files), rolling it back needs them as well: they redeploy the files of the previous tag.
A new image
- Change
deploy/container/text_lab.defand build the image on a compute node. - Copy it to
$TEXT_LAB_BASE/container/under a new name (keep the old image: running sessions use it). - Run the tests with the new image (
TL_CONTAINERinscripts/test_on_node.sbatch) and try it in a sandbox app (TL_CONTAINERindev.env). - Set
TL_CONTAINERinsite.env, commit, and copysite.envto its live location. Rolling back means restoring the previous value.
First release of this layout (3.0.0)
Versions before 3.0.0 have the path of their code
($TEXT_LAB_BASE/src_main280526/src) in the deployed launch script and
read no site configuration. Version 3.0.0 is the first to read site.env
and serve current, so the switch needs the HPC team once:
- Prepare the shared folder without touching the running deployment: the
release folder
releases/v3.0.0,currentpointing to it, andsite.env. The old launch script reads none of them. - Rehearse in a sandbox app without
dev.env: it then runs exactly what production will run. - Ask the HPC team to deploy the Open OnDemand files from tag
v3.0.0. Merge tomainclose to that date, since merging publishes the new user guide. - Start a session on the production app and check
output.logand each feature. - To roll back, the HPC team redeploys the files of
v2.2.1, which still servesrc_main280526; keep that folder until 3.0.0 has run without problems for a while. - Afterwards, remove
src_main280526and the old versions' session logs (logs/).ollama_tmp/andrun_olmocr.shin the shared folder are used by no version in the repository; check with their owners before removing them.