# BLIP-2 Projects at AI Tinkerers

> Canonical HTML: https://aitinkerers.org/technologies/blip-2
> Markdown URL: https://aitinkerers.org/technologies/blip-2.md
> Technology record last updated: 2026-02-25T04:12:46Z
> Generated: 2026-09-22T08:38:52Z

Salesforce Research's BLIP-2 uses a Q-Former to bridge frozen image encoders and large language models for high-efficiency multimodal reasoning.

Salesforce Research built BLIP-2 to eliminate the high compute costs of traditional vision-language training. The architecture uses a Q-Former (Querying Transformer) to connect a frozen image encoder (ViT-L/14) with a frozen LLM (Flan-T5 or OPT). This method achieves state-of-the-art zero-shot results on visual question answering (VQA) and image captioning. With only 188 million trainable parameters, BLIP-2 beats the 80-billion-parameter Flamingo model on zero-shot VQA v2 benchmarks. It provides a 54x reduction in trainable parameters while maintaining top-tier multimodal performance.

- Official technology site: https://huggingface.co/docs/transformers/model_doc/blip-2
- Public AI Tinkerers demos and talks: 3
- Result page: 1 of 1

## Recent Public Talks and Demos

### [classifai.dev - simple, self-improving classification api](https://la.aitinkerers.org/talks/rsvp_1OwmludKh3w)

The simplest possible classification API that any developer can use. Simply provide data you'd like classified, and the classes/labels OR a description of the task. If you provide feedback, the same endpoint starts improving its responses over time.

- Event context: AI Tinkerers LA – October 2025: Ghosts in the Machine w/ Oxen.ai — 2025-10-21 — Los Angeles
- Public talk page: https://la.aitinkerers.org/talks/rsvp_1OwmludKh3w

### [Browsing the web with AI](https://toronto.aitinkerers.org/talks/rsvp_mtJyKffdKIM)

Web scraping is broken. Companies spend millions maintaining brittle scrapers while developers waste countless hours rebuilding the same solutions. The emergence of powerful vision-language models (VLMs) and LLMs creates an opportunity to revolutionize this space. I'll demonstrate a novel architecture that combines: - Microsoft VLM for visual understanding and DOM parsing - Claude for reasoning and task planning - Selenium for browser automation - Custom prompt engineering for reliable structured output We'll explore: - Why traditional scrapers fail - How VLMs understand web interfaces - Prompt engineering for reliable agents - Live demo: Flight price comparison - Challenges in hallucination prevention - Open source architecture decisions Key technical innovations: - Vision-guided DOM traversal - RAG memory during browsing - Structured data extraction This project started from personal frustration with repetitive research tasks. The goal: make web automation accessible to everyone while being reliable enough for production use. Live demo will showcase the agent finding flight prices and returning structured JSON - all without human intervention.

- Event context: AI Tinkerers Toronto - January 2025 Meetup at Google **sold out** — 2025-01-30 — Toronto
- Public talk page: https://toronto.aitinkerers.org/talks/rsvp_mtJyKffdKIM

### [Applying 4o Vision Finetuning to Chemistry Diagrams](https://singapore.aitinkerers.org/talks/rsvp_7w9joR3W_oI)

The task is to extract student's attempts for chemistry diagram questions. These diagrams are a graph with nodes and edges. Using VLLMs out of the box often results in the model correcting the chemistry equations or missing key notation. Here we explore Vision Finetuning, and see how far we can go with less than 10 hand-labelled examples. Kuang Wen and I will show the data we have, the augmentation techniques, and our current demo app comparing finetuned with non-finetuned.

- Event context: AI Tinkerers Singapore: 3rd Meetup - November 19th, 2024 — 2024-11-19 — Singapore
- Public talk page: https://singapore.aitinkerers.org/talks/rsvp_7w9joR3W_oI

## Related Technologies

- [BLIP](https://aitinkerers.org/technologies/blip) ([Markdown](https://aitinkerers.org/technologies/blip.md)) — 4 public demos
- [CLIP](https://aitinkerers.org/technologies/clip) ([Markdown](https://aitinkerers.org/technologies/clip.md)) — 10 public demos
- [Flamingo](https://aitinkerers.org/technologies/flamingo) ([Markdown](https://aitinkerers.org/technologies/flamingo.md)) — 3 public demos
- [LXMERT](https://aitinkerers.org/technologies/lxmert) ([Markdown](https://aitinkerers.org/technologies/lxmert.md)) — 4 public demos
- [UNITER](https://aitinkerers.org/technologies/uniter) ([Markdown](https://aitinkerers.org/technologies/uniter.md)) — 3 public demos
- [ViLBERT](https://aitinkerers.org/technologies/vilbert) ([Markdown](https://aitinkerers.org/technologies/vilbert.md)) — 4 public demos
- [VisualBERT](https://aitinkerers.org/technologies/visualbert) ([Markdown](https://aitinkerers.org/technologies/visualbert.md)) — 3 public demos
- [ABBYY FineReader](https://aitinkerers.org/technologies/abbyy-finereader) ([Markdown](https://aitinkerers.org/technologies/abbyy-finereader.md)) — 3 public demos
- [Amazon Textract](https://aitinkerers.org/technologies/amazon-textract) ([Markdown](https://aitinkerers.org/technologies/amazon-textract.md)) — 5 public demos
- [Azure Computer Vision](https://aitinkerers.org/technologies/azure-computer-vision) ([Markdown](https://aitinkerers.org/technologies/azure-computer-vision.md)) — 1 public demo
- [BERT](https://aitinkerers.org/technologies/bert) ([Markdown](https://aitinkerers.org/technologies/bert.md)) — 179 public demos
- [BLOOM](https://aitinkerers.org/technologies/bloom) ([Markdown](https://aitinkerers.org/technologies/bloom.md)) — 115 public demos
- [Claude](https://aitinkerers.org/technologies/claude) ([Markdown](https://aitinkerers.org/technologies/claude.md)) — 174 public demos
- [Data Augmentation](https://aitinkerers.org/technologies/data-augmentation) ([Markdown](https://aitinkerers.org/technologies/data-augmentation.md)) — 1 public demo
- [Demo App](https://aitinkerers.org/technologies/demo-app) ([Markdown](https://aitinkerers.org/technologies/demo-app.md)) — 1 public demo
- [DOM parsing](https://aitinkerers.org/technologies/dom-parsing) ([Markdown](https://aitinkerers.org/technologies/dom-parsing.md)) — 1 public demo
- [Fireworks AI](https://aitinkerers.org/technologies/fireworks-ai) ([Markdown](https://aitinkerers.org/technologies/fireworks-ai.md)) — 1 public demo
- [Google Cloud Vision API](https://aitinkerers.org/technologies/google-cloud-vision-api) ([Markdown](https://aitinkerers.org/technologies/google-cloud-vision-api.md)) — 1 public demo