# CLIP Projects at AI Tinkerers

> Canonical HTML: https://aitinkerers.org/technologies/clip
> Markdown URL: https://aitinkerers.org/technologies/clip.md
> Technology record last updated: 2026-02-25T04:10:54Z
> Generated: 2026-08-26T09:16:45Z

CLIP (Contrastive Language–Image Pre-training) is an OpenAI neural network that connects visual and textual data for powerful zero-shot image classification.

CLIP, developed by OpenAI, is a multimodal AI model: it learns visual concepts directly from natural language supervision. The system trains a text encoder and an image encoder to predict which text-image pairs match within a massive dataset (400 million pairs). This contrastive pre-training eliminates the need for expensive, manually labeled datasets like ImageNet. The key capability is zero-shot transfer: the model can classify an image into any category, such as 'a photo of a vintage motorcycle,' without explicit, task-specific training data.

- Official technology site: https://openai.com/blog/clip/
- Public AI Tinkerers demos and talks: 10
- Result page: 1 of 1

## Recent Public Talks and Demos

### [Shop Talk](https://st-louis.aitinkerers.org/talks/rsvp_HCVQpW0TgEw)

I built ShopTalk, a multimodal shopping assistant that lets users search for products using text, images, or a combination of both, delivering more human-like and relevant results. In the demo, users can upload an image, type a query, or do both, and the system uses BLIP to generate contextual captions and CLIP to embed both text and images into a shared vector space, retrieving and ranking the most relevant products through a Redis-backed similarity search with late fusion scoring.

- Event context: AI Tinkerers St. Louis: April 2026 Meetup — 2026-04-14 — St. Louis
- Public talk page: https://st-louis.aitinkerers.org/talks/rsvp_HCVQpW0TgEw

### [Let's talk about Embeddings](https://cologne.aitinkerers.org/talks/rsvp_NX6I1fvuINY)

I will talk about why embeddings are such a great thing. They can do so many tasks that we set out a huge LLM to do, but in a much more efficient and cost saving way. There are tons of use cases for embeddings, and in this talk, I just want to give a simple insight into some use cases of embeddings, beside RAG. I want to cover (not sure if this is the final list yet): - RAG - Image Search - Image Classifier - Advanced Image Classifier with an added MLP Head - Text Matching across languages - Getting Clear Text Input for Customer Intention Analysis (Main Focus) - And a short example of how you can use that clear text input to improve what you are offering as a company. (Main Focus) As the 5 Minute Time slot is very narrow, I will likely focus on the Clear Text Input Analysis part, as I think that is quite a nice use case for embedding based, customer facing search. While I will not show a lot of code in this presentation, coding this yourself is so easy, that anyone could do it without seeing any code. It's more about the idea and concept for this usecase.

- Event context: AI Tinkerers Cologne 3: Demos, Code, and Architecture — 2026-03-05 — Cologne
- Public talk page: https://cologne.aitinkerers.org/talks/rsvp_NX6I1fvuINY

### [How we made our RAG truly multimodal](https://tokyo.aitinkerers.org/talks/rsvp_xQClQ7BcKJ4)

While text-based RAG is a solved problem, building a truly multimodal retrieval pipeline remains the Wild West. At Archie, we needed our AI to understand not just code repositories, but the visual context of architecture diagrams and screenshots. In this talk, I will share the engineering journey of building a multimodal RAG system from scratch when no tutorials existed. I will cover: Data Ingestion: How to process and chunk distinct modalities (images vs. text) effectively. Embedding Strategies: Comparing CLIP vs. newer multimodal embedding models—and what actually worked for technical diagrams. Retrieval Architecture: How we structured our vector search to perform hybrid retrieval (text + image) to ground the MLLM in the correct context. The "Gotchas": Specific failures we encountered when trying to scale vision-based retrieval.

- Event context: AI Tinkerers Ginza Meetup – January 15, 2026 — 2026-01-15 — Tokyo
- Public talk page: https://tokyo.aitinkerers.org/talks/rsvp_xQClQ7BcKJ4

### [classifai.dev - simple, self-improving classification api](https://la.aitinkerers.org/talks/rsvp_1OwmludKh3w)

The simplest possible classification API that any developer can use. Simply provide data you'd like classified, and the classes/labels OR a description of the task. If you provide feedback, the same endpoint starts improving its responses over time.

- Event context: AI Tinkerers LA – October 2025: Ghosts in the Machine w/ Oxen.ai — 2025-10-21 — Los Angeles
- Public talk page: https://la.aitinkerers.org/talks/rsvp_1OwmludKh3w

### [A Google-Like Search Experience for Your Photo Library with Opensource Tools](https://raleigh.aitinkerers.org/talks/rsvp_EEHGIBjFDGQ)

This demo showcases a photo album curation workflow using FiftyOne’s Brain similarity indexing and multimodal search. Images are embedded and indexed once via compute_similarity(), enabling rapid retrieval of visually similar photos and natural language search (e.g., “sunset at the beach”) using CLIP, which aligns text and images in the same embedding space. Users can sort albums by similarity to a reference image or a text prompt, find duplicates, and surface on-theme groups for curation. Low‑dimensional embedding visualizations further reveal clusters and outliers that guide pruning and organization. The experience is fully interactive in the FiftyOne App, and scalable via vector database integrations when working with large libraries.

- Event context: AI Tinkerers - Raleigh Inaugural Meetup (September 2025) — 2025-09-30 — Raleigh
- Public talk page: https://raleigh.aitinkerers.org/talks/rsvp_EEHGIBjFDGQ

### [Multimodalidad en espacios vectoriales](https://bogota.aitinkerers.org/talks/rsvp_dSbIR9jaxqw)

En esta charla, exploraremos cómo utilizar embeddings multimodales para representar información de distintas fuentes (texto, imágenes, audio) en un mismo espacio vectorial. A través de código y casos prácticos, veremos cómo modelos como CLIP y CLAP permiten crear conexiones entre texto e imágenes o texto y audio. Se analizará la arquitectura subyacente, incluyendo el proceso de generación de embeddings, métricas para evaluar la calidad de las representaciones y cómo aplicar estos modelos en tareas como búsqueda semántica y clasificación multimodal.

- Event context: AI Tinkerers Bogotá - Febrero :) — 2025-02-27 — Bogotá
- Public talk page: https://bogota.aitinkerers.org/talks/rsvp_dSbIR9jaxqw

### [Browsing the web with AI](https://toronto.aitinkerers.org/talks/rsvp_mtJyKffdKIM)

Web scraping is broken. Companies spend millions maintaining brittle scrapers while developers waste countless hours rebuilding the same solutions. The emergence of powerful vision-language models (VLMs) and LLMs creates an opportunity to revolutionize this space. I'll demonstrate a novel architecture that combines: - Microsoft VLM for visual understanding and DOM parsing - Claude for reasoning and task planning - Selenium for browser automation - Custom prompt engineering for reliable structured output We'll explore: - Why traditional scrapers fail - How VLMs understand web interfaces - Prompt engineering for reliable agents - Live demo: Flight price comparison - Challenges in hallucination prevention - Open source architecture decisions Key technical innovations: - Vision-guided DOM traversal - RAG memory during browsing - Structured data extraction This project started from personal frustration with repetitive research tasks. The goal: make web automation accessible to everyone while being reliable enough for production use. Live demo will showcase the agent finding flight prices and returning structured JSON - all without human intervention.

- Event context: AI Tinkerers Toronto - January 2025 Meetup at Google **sold out** — 2025-01-30 — Toronto
- Public talk page: https://toronto.aitinkerers.org/talks/rsvp_mtJyKffdKIM

### [Applying 4o Vision Finetuning to Chemistry Diagrams](https://singapore.aitinkerers.org/talks/rsvp_7w9joR3W_oI)

The task is to extract student's attempts for chemistry diagram questions. These diagrams are a graph with nodes and edges. Using VLLMs out of the box often results in the model correcting the chemistry equations or missing key notation. Here we explore Vision Finetuning, and see how far we can go with less than 10 hand-labelled examples. Kuang Wen and I will show the data we have, the augmentation techniques, and our current demo app comparing finetuned with non-finetuned.

- Event context: AI Tinkerers Singapore: 3rd Meetup - November 19th, 2024 — 2024-11-19 — Singapore
- Public talk page: https://singapore.aitinkerers.org/talks/rsvp_7w9joR3W_oI

### [Visual prompting for fun and profit](https://portland.aitinkerers.org/talks/rsvp_fGAlJQAvWUA)

Cloud hosted LLMs (such as GPT4 by OpenAI) charge per-token. Depending on the length of your input, this can become expensive, if not impossible. When building RealEstatePulse, I came across the challenge of needing to extract structured data from city zoning codes that were hundreds of pages long -- well passed the capabilities of even the largest LLMs. Moreover, there was a heavy dependence on textual layout (for example, with tables). The extra HTML markup blew up the token count and made analyzing zoning codes very expensive. However, recent research in LLMs have shown that the world model they acquire from textual learning translates well into visual question answering. Moreover, there is good reason to believe that current models are highly redundant, meaning they can theoretically process a lot more data than what the token embedding layer produces. Some cloud LLMs, such as GPT4, offer the option of visual input. Unlike text input, visual input is charged at a flat fee depending on the size of the image. I would like to present the technique I developed for analyzing extremely large zoning codes by rendering the text as an image and using this to prompt GPT. I will show that this makes the problem tractable and produces good results.

- Event context: AI Tinkerers Portland Inaugural Meetup - July — 2024-07-24 — Portland
- Public talk page: https://portland.aitinkerers.org/talks/rsvp_fGAlJQAvWUA

### [autodistill - an open source framework for model distillation](https://sf.aitinkerers.org/talks/rsvp_bkDN_zUQGV4)

autodistill is an open source framework for distilling big, general, slower models into domain-specific, smaller, and faster models. Right now, it best supports computer vision tasks. With autodistill, users apply foundation models like SAM and CLIP to auto label datasets and train smaller models in fewer than 10 lines of code. Hackers want to adapt foundation models to domain-specific models that they own, often leveraging their proprietary data. autodistill presents one open source approach to this.

- Event context: 🤖🔄🧠 AI Tinkerers SF - August Meetup — 2023-08-10 — San Francisco
- Public talk page: https://sf.aitinkerers.org/talks/rsvp_bkDN_zUQGV4

## Related Technologies

- [BLIP](https://aitinkerers.org/technologies/blip) ([Markdown](https://aitinkerers.org/technologies/blip.md)) — 4 public demos
- [LXMERT](https://aitinkerers.org/technologies/lxmert) ([Markdown](https://aitinkerers.org/technologies/lxmert.md)) — 4 public demos
- [ViLBERT](https://aitinkerers.org/technologies/vilbert) ([Markdown](https://aitinkerers.org/technologies/vilbert.md)) — 4 public demos
- [BLIP-2](https://aitinkerers.org/technologies/blip-2) ([Markdown](https://aitinkerers.org/technologies/blip-2.md)) — 3 public demos
- [Flamingo](https://aitinkerers.org/technologies/flamingo) ([Markdown](https://aitinkerers.org/technologies/flamingo.md)) — 3 public demos
- [UNITER](https://aitinkerers.org/technologies/uniter) ([Markdown](https://aitinkerers.org/technologies/uniter.md)) — 3 public demos
- [VisualBERT](https://aitinkerers.org/technologies/visualbert) ([Markdown](https://aitinkerers.org/technologies/visualbert.md)) — 3 public demos
- [GPT-4](https://aitinkerers.org/technologies/gpt-4) ([Markdown](https://aitinkerers.org/technologies/gpt-4.md)) — 529 public demos
- [OpenAI](https://aitinkerers.org/technologies/openai) ([Markdown](https://aitinkerers.org/technologies/openai.md)) — 111 public demos
- [Pinecone](https://aitinkerers.org/technologies/pinecone) ([Markdown](https://aitinkerers.org/technologies/pinecone.md)) — 25 public demos
- [Python](https://aitinkerers.org/technologies/python) ([Markdown](https://aitinkerers.org/technologies/python.md)) — 654 public demos
- [PyTorch](https://aitinkerers.org/technologies/pytorch) ([Markdown](https://aitinkerers.org/technologies/pytorch.md)) — 273 public demos
- [RAG](https://aitinkerers.org/technologies/rag) ([Markdown](https://aitinkerers.org/technologies/rag.md)) — 146 public demos
- [Transformers](https://aitinkerers.org/technologies/transformers) ([Markdown](https://aitinkerers.org/technologies/transformers.md)) — 148 public demos
- [ABBYY FineReader](https://aitinkerers.org/technologies/abbyy-finereader) ([Markdown](https://aitinkerers.org/technologies/abbyy-finereader.md)) — 3 public demos
- [Amazon Textract](https://aitinkerers.org/technologies/amazon-textract) ([Markdown](https://aitinkerers.org/technologies/amazon-textract.md)) — 5 public demos
- [autodistill](https://aitinkerers.org/technologies/autodistill) ([Markdown](https://aitinkerers.org/technologies/autodistill.md)) — 1 public demo
- [Azure Computer Vision](https://aitinkerers.org/technologies/azure-computer-vision) ([Markdown](https://aitinkerers.org/technologies/azure-computer-vision.md)) — 1 public demo