[sal-uva]

Changes:

Embedding support in back-end

LLM client

• Abstract embed(), filled in by openai_client and ollama_client. This could've used LangChain but it doesn't support multimodal embeddings yet...
• Fixes and changes in what a model supports (input and output). This was broken before.
• Models are now classified as embedding or generative, and they should be filtered in llm_prompter and the embedding processors. It uses Ollama for model capabilities or else checks whether "embed" appears in the model ID, which isn't great but there seems to be no clear endpoint for embedding capacities yet...
• For LLM server installation, accept both "Authorization: bearer" and just the bearer as auth
• Make sure that only models that are reachable by a particular user are listed in processor options. Model lookup now happens in one place (resolve_model), so the llm.access privilege is applied everywhere. Applied to llm_prompter and embedding processors.
• Write embeddings as annotations (but don't show in the Explorer).

Ollama

• Support embeddings through the /embed endpoint. [There's quite a few embedding models on Ollama now!](https://ollama.com/search?c=embedding)
• Fetch what input type a model supports. This now also includes 'embedding'
• Define what output type a model supports: 'generate' or 'embed'.
• Multimodal embeddings (needs Ollama >= 0.40)
• Caveats:
  • Model info via api/show is not always reliable (not our fault).
  • No video is supported yet. This is 'solved' in embed_media by just taking a frame for now.
  • embed() can move to an OpenAI-compatible endpoint once Ollama does this, which they say they're working on.

OpenAI / vLLM

• Add feature to check for available models. This is a bit messy since vLLM and OpenAI don't always agree (e.g. is it /api/models or just /models). I decided it's better to keep them as one client anyway since there is a shared convention and maybe things will converge in the future.
• Add /embed. Again, with some scaffolding to make sure the input format is correct with different modalities (text, image, video, etc).
• Embedding truncation (Matryoshka models) and a fix for instructions not being sent along with media-type embeddings.

New processors

Generating:

• embed_text.py: one vector per item.
• embed_media.py:
  • abstract base for embed_images.py, embed_videos.py and embed_audio.py.
  • Files are compressed before sending (Pillow for images, ffmpeg for video/audio) since they travel base64 inside the request. This makes the processors a bit bloaty but the alternative was to have a long chain of processors before you can reach the embedder.
  • Fails after 10 consecutive failures rather than 10 in total, and pads frames so 1-frame videos don't break.

Analysing:

• reduce_embeddings.py: reduce to N dimensions with UMAP, t-SNE or PCA.
• cluster_embeddings.py: cluster with HDBSCAN or k-means.
  • Mimicks colink_urls's threading to prevent using 100% memory (capped at 90% for now).
  • Can write different clustering metrics as annotations (cluster number, probability, and concentration)
  • This also can reduce the dimensions of embeddings which makes it overlap with the Reduce embeddings processor. I chose to keep this separate anyway out of convenience and because they share functions. The cluster processor can also still be ran on the output of the reduce embedding processor.
• embedding_similarity.py: scores every item against a text you give it, embedded with the same model as the parent.
• latent_space_path.py: walks from one embedding to its nearest unvisited neighbour, on and on. Neighbourhood capped at 50k.

Plotting:

• embedding_map.py / embedding_map_3d.py: interactive HTML scatterplot of the reduced embeddings, canvas-rendered with pan/zoom and hover, self-contained with no external dependencies. Colours by cluster and can draw thumbnails instead of dots.
• embedding_streams.py: plots distance from a reference embedding over time, to show how 'stable' a set of embeddings stays. This works needs a bit of love still--to do!

Media helpers:

• media_thumbnails.py: square thumbnails for a media archive, with an optional column treshold filter.
• media_sprite.py: merges thumbnails into one sprite sheet plus a metadata file with each image's position. Used by the embedding plots to put images on the scatterplot wihtout having to load all images separately.

Memory

• New embeddings.max_embeddings admin setting that caps how many embeddings one processor reads at once (default 500k, '0' for no limit). Applied to the generating and analysing processors.
• The processors that hold a full embedding matrix are in one job queue, so only one runs at a time.
• Shared helpers in common/lib/embeddings.py read vectors in batches.
• embedding_similarity sorts via a scratch file, so only each item's score and offset stay in memory rather than its text.
• The thumbnail and sprite processors don't keep every image in memory: thumbnails go straight into the zip, and the sprite reads sizes from headers first and then decodes one picture at a time.
  • the sprite sheet itself still scales with the number of tiles.

Other stuff

• Do a better job at showing LLM capacities in model list. Refer to model card if no capacities are set in llms.json for third-party. llms.json now also declares supported_media_types per model.
• Use the datasource __init__.py to set an Explorer template and use this for all TikTok data sources (minus comments). Needs to be checked after redesign (where I added 'variants').
• Changes to TikTok Explorer template: videos now autoplay and you can scroll down with buttons (mimicking the TikTok brainrot 'experience')
• Video (scene) frame extractors now write a proper metadata.json and declare media_type = image, so the thumbnail/sprite chain works on them.
• HTML previews open as utf-8 (was breaking on Windows).
• New dependencies: umap-learn, and scikit-learn>=1.3 for HDBSCAN.
• Tests tests tests for the embeddings, the processors and the annotation writing.

questions/comments/stuff to think about

• the /embed endpoint through vLLM is not super reliable yet--there are not a lot of multimodal models beyond the Qwen3-VL one. I'll keep an eye out and update update when things have settled
• In the future it would probably make sense to have embeddings as an output / 'media' type so that post-processors can easily run on it. And it'd be great to be able to import embedding datasets as well.
• Telling embedding models from generative ones is still not robust. The name heuristic misses embedding models that aren't named for it (bge-m3, all-minilm). LiteLLM has some indication of this but it's still in flux.
• The memory watchdog in the clustering processor reads host memory, so it won't fire if the container has its own memory limit, and it only runs where fork is available (so not on Windows, but who cares about Windows?).
• media_sprite.py processor: I'm doubting whether this can have a better spot. First I thought it could be merged with the image wall, but the output of the sprite needs to be a zip with the image and a metadata file listing the position of images so I kept them separate. But, it could also be integrated into the embedding plotters. Ultimately I think this is the right spot though. Maybe we want to use spritesheets as efficient thumbnail lookups in more visualizations in the future as well.

Tests

Tested with an SRC server running vLLM with Qwen3-VL-Embedding-8B (note: --runner pooling is required, otherwise the embeddings endpoint is not served at all), and a local Ollama with mxbai-embed-large and embeddinggemma-2.

[sal-uva]

Updated, I think this is ready to merge!