# Set of Marks Projects at AI Tinkerers

> Canonical HTML: https://aitinkerers.org/technologies/set-of-marks
> Markdown URL: https://aitinkerers.org/technologies/set-of-marks.md
> Technology record last updated: 2026-03-03T01:10:32Z
> Generated: 2026-09-23T07:37:58Z

Set-of-Mark (SoM) is a visual prompting technique: it segments images and overlays alphanumeric marks to dramatically enhance Large Multimodal Models' (LMMs) visual grounding capabilities.

SoM is a novel visual prompting method designed to unleash the full visual grounding potential of LMMs, specifically GPT-4V. The process is direct: an off-the-shelf segmentation model (e.g., SEEM or SAM) partitions an image into distinct regions. We then overlay these regions with a set of speakable marks (alphanumerics, masks, or boxes). This marked image, when input to the LMM, provides explicit spatial and object relationship context, which was previously a blind spot. Empirical studies confirm SoM's effectiveness: it enables GPT-4V to outperform state-of-the-art, fully-finetuned models on fine-grained vision tasks like RefCOCOg, all in a zero-shot setting.

- Official technology site: https://github.com/microsoft/SoM
- Public AI Tinkerers demos and talks: 1
- Result page: 1 of 1

## Recent Public Talks and Demos

### [Multimodal Maestro with GPT-4 Vision](https://sf.aitinkerers.org/talks/rsvp_5dtCu0fp6a8)

Multimodal-Maestro gives you more control over large multimodal models to get the outputs you want. With more effective prompting tactics, you can get multimodal models to do tasks you didn't know (or think!) were possible. Additionally, GPT-4 vision and its open source alternatives like CogLVM and LLaVa lack grounding capabilities out-of-the-box. This demo explains how emerging techniques like Set of Marks can be used in tandem with these large models to improve capabilities on vision tasks. Specifically, we'll be demoing how to identify and draw a mask over a dog using segment anything guided marks and gpt-4-vision.

- Event context: AI Tinkerers SF X Cerebral Valley - Dec Meetup — 2023-12-04 — San Francisco
- Public talk page: https://sf.aitinkerers.org/talks/rsvp_5dtCu0fp6a8

## Related Technologies

- [CogLVM](https://aitinkerers.org/technologies/coglvm) ([Markdown](https://aitinkerers.org/technologies/coglvm.md)) — 1 public demo
- [GPT-4 Vision](https://aitinkerers.org/technologies/gpt-4-vision) ([Markdown](https://aitinkerers.org/technologies/gpt-4-vision.md)) — 2 public demos
- [LLaVA](https://aitinkerers.org/technologies/llava) ([Markdown](https://aitinkerers.org/technologies/llava.md)) — 5 public demos
- [Segment Anything Model](https://aitinkerers.org/technologies/segment-anything-model) ([Markdown](https://aitinkerers.org/technologies/segment-anything-model.md)) — 5 public demos