# ViLBERT Projects at AI Tinkerers

> Canonical HTML: https://aitinkerers.org/technologies/vilbert
> Markdown URL: https://aitinkerers.org/technologies/vilbert.md
> Technology record last updated: 2026-02-25T04:11:11Z
> Generated: 2026-09-21T05:37:07Z

ViLBERT adapts the BERT architecture to process visual and textual data simultaneously through a dual-stream transformer.

Researchers from Georgia Tech and Facebook AI Research (FAIR) built ViLBERT to master multimodal tasks. The system employs a two-stream architecture to process visual regions and text tokens separately before merging them through co-attentional transformer layers. This setup enables the model to learn complex relationships between images and language. ViLBERT established state-of-the-art results on the VQA 2.0 and VCR datasets (Visual Commonsense Reasoning) upon its release. It leverages pre-training on the Conceptual Captions dataset (3.3 million image-caption pairs) to develop robust, task-agnostic representations.

- Official technology site: https://arxiv.org/abs/1908.02265
- Public AI Tinkerers demos and talks: 4
- Result page: 1 of 1

## Recent Public Talks and Demos

### [classifai.dev - simple, self-improving classification api](https://la.aitinkerers.org/talks/rsvp_1OwmludKh3w)

The simplest possible classification API that any developer can use. Simply provide data you'd like classified, and the classes/labels OR a description of the task. If you provide feedback, the same endpoint starts improving its responses over time.

- Event context: AI Tinkerers LA – October 2025: Ghosts in the Machine w/ Oxen.ai — 2025-10-21 — Los Angeles
- Public talk page: https://la.aitinkerers.org/talks/rsvp_1OwmludKh3w

### [Browsing the web with AI](https://toronto.aitinkerers.org/talks/rsvp_mtJyKffdKIM)

Web scraping is broken. Companies spend millions maintaining brittle scrapers while developers waste countless hours rebuilding the same solutions. The emergence of powerful vision-language models (VLMs) and LLMs creates an opportunity to revolutionize this space. I'll demonstrate a novel architecture that combines: - Microsoft VLM for visual understanding and DOM parsing - Claude for reasoning and task planning - Selenium for browser automation - Custom prompt engineering for reliable structured output We'll explore: - Why traditional scrapers fail - How VLMs understand web interfaces - Prompt engineering for reliable agents - Live demo: Flight price comparison - Challenges in hallucination prevention - Open source architecture decisions Key technical innovations: - Vision-guided DOM traversal - RAG memory during browsing - Structured data extraction This project started from personal frustration with repetitive research tasks. The goal: make web automation accessible to everyone while being reliable enough for production use. Live demo will showcase the agent finding flight prices and returning structured JSON - all without human intervention.

- Event context: AI Tinkerers Toronto - January 2025 Meetup at Google **sold out** — 2025-01-30 — Toronto
- Public talk page: https://toronto.aitinkerers.org/talks/rsvp_mtJyKffdKIM

### [Applying 4o Vision Finetuning to Chemistry Diagrams](https://singapore.aitinkerers.org/talks/rsvp_7w9joR3W_oI)

The task is to extract student's attempts for chemistry diagram questions. These diagrams are a graph with nodes and edges. Using VLLMs out of the box often results in the model correcting the chemistry equations or missing key notation. Here we explore Vision Finetuning, and see how far we can go with less than 10 hand-labelled examples. Kuang Wen and I will show the data we have, the augmentation techniques, and our current demo app comparing finetuned with non-finetuned.

- Event context: AI Tinkerers Singapore: 3rd Meetup - November 19th, 2024 — 2024-11-19 — Singapore
- Public talk page: https://singapore.aitinkerers.org/talks/rsvp_7w9joR3W_oI

### [Visual prompting for fun and profit](https://portland.aitinkerers.org/talks/rsvp_fGAlJQAvWUA)

Cloud hosted LLMs (such as GPT4 by OpenAI) charge per-token. Depending on the length of your input, this can become expensive, if not impossible. When building RealEstatePulse, I came across the challenge of needing to extract structured data from city zoning codes that were hundreds of pages long -- well passed the capabilities of even the largest LLMs. Moreover, there was a heavy dependence on textual layout (for example, with tables). The extra HTML markup blew up the token count and made analyzing zoning codes very expensive. However, recent research in LLMs have shown that the world model they acquire from textual learning translates well into visual question answering. Moreover, there is good reason to believe that current models are highly redundant, meaning they can theoretically process a lot more data than what the token embedding layer produces. Some cloud LLMs, such as GPT4, offer the option of visual input. Unlike text input, visual input is charged at a flat fee depending on the size of the image. I would like to present the technique I developed for analyzing extremely large zoning codes by rendering the text as an image and using this to prompt GPT. I will show that this makes the problem tractable and produces good results.

- Event context: AI Tinkerers Portland Inaugural Meetup - July — 2024-07-24 — Portland
- Public talk page: https://portland.aitinkerers.org/talks/rsvp_fGAlJQAvWUA

## Related Technologies

- [CLIP](https://aitinkerers.org/technologies/clip) ([Markdown](https://aitinkerers.org/technologies/clip.md)) — 10 public demos
- [LXMERT](https://aitinkerers.org/technologies/lxmert) ([Markdown](https://aitinkerers.org/technologies/lxmert.md)) — 4 public demos
- [BLIP](https://aitinkerers.org/technologies/blip) ([Markdown](https://aitinkerers.org/technologies/blip.md)) — 4 public demos
- [BLIP-2](https://aitinkerers.org/technologies/blip-2) ([Markdown](https://aitinkerers.org/technologies/blip-2.md)) — 3 public demos
- [Flamingo](https://aitinkerers.org/technologies/flamingo) ([Markdown](https://aitinkerers.org/technologies/flamingo.md)) — 3 public demos
- [UNITER](https://aitinkerers.org/technologies/uniter) ([Markdown](https://aitinkerers.org/technologies/uniter.md)) — 3 public demos
- [VisualBERT](https://aitinkerers.org/technologies/visualbert) ([Markdown](https://aitinkerers.org/technologies/visualbert.md)) — 3 public demos
- [GPT-4](https://aitinkerers.org/technologies/gpt-4) ([Markdown](https://aitinkerers.org/technologies/gpt-4.md)) — 529 public demos
- [ABBYY FineReader](https://aitinkerers.org/technologies/abbyy-finereader) ([Markdown](https://aitinkerers.org/technologies/abbyy-finereader.md)) — 3 public demos
- [Amazon Textract](https://aitinkerers.org/technologies/amazon-textract) ([Markdown](https://aitinkerers.org/technologies/amazon-textract.md)) — 5 public demos
- [Azure Computer Vision](https://aitinkerers.org/technologies/azure-computer-vision) ([Markdown](https://aitinkerers.org/technologies/azure-computer-vision.md)) — 1 public demo
- [BAN](https://aitinkerers.org/technologies/ban) ([Markdown](https://aitinkerers.org/technologies/ban.md)) — 1 public demo
- [BERT](https://aitinkerers.org/technologies/bert) ([Markdown](https://aitinkerers.org/technologies/bert.md)) — 179 public demos
- [BLOOM](https://aitinkerers.org/technologies/bloom) ([Markdown](https://aitinkerers.org/technologies/bloom.md)) — 115 public demos
- [Claude](https://aitinkerers.org/technologies/claude) ([Markdown](https://aitinkerers.org/technologies/claude.md)) — 173 public demos
- [Data Augmentation](https://aitinkerers.org/technologies/data-augmentation) ([Markdown](https://aitinkerers.org/technologies/data-augmentation.md)) — 1 public demo
- [Demo App](https://aitinkerers.org/technologies/demo-app) ([Markdown](https://aitinkerers.org/technologies/demo-app.md)) — 1 public demo
- [DOM parsing](https://aitinkerers.org/technologies/dom-parsing) ([Markdown](https://aitinkerers.org/technologies/dom-parsing.md)) — 1 public demo