# wav2vec 2 Projects at AI Tinkerers

> Canonical HTML: https://aitinkerers.org/technologies/wav2vec-2
> Markdown URL: https://aitinkerers.org/technologies/wav2vec-2.md
> Technology record last updated: 2026-05-25T08:22:15Z
> Generated: 2026-09-23T07:41:01Z

A self-supervised framework that learns powerful speech representations from raw audio, drastically reducing the need for labeled training data.

Developed by Meta AI, wav2vec 2.0 processes raw audio through a convolutional encoder to construct latent representations, which are then masked and passed to a Transformer network. By solving a contrastive task over a quantized vocabulary of speech units, the model learns the underlying structure of spoken language without relying on manual transcriptions. This self-supervised approach allows developers to build highly accurate speech recognition systems: fine-tuning on just 10 minutes of labeled data yields a competitive 4.8% Word Error Rate (WER) on the LibriSpeech clean test set.

- Official technology site: https://arxiv.org/abs/2006.11477
- Public AI Tinkerers demos and talks: 2
- Result page: 1 of 1

## Recent Public Talks and Demos

### [Speak mk1: A multimodal mamba-attention hybrid model for speech therapy](https://dubai.aitinkerers.org/talks/rsvp_f9s0FLsNTT0)

I am currently building a mamba-attention hybrid encoder and decoder from scratch for speech therapy, not fine-tuning, not wrapping an API, but training every component end to end on my RTX 4060 laptop. The system has three components I am building in parallel: The first is a custom Mamba SSM-based audio encoder trained on LibriSpeech with multi-task phonological heads for voicing, manner, place of articulation, and correctness detection. The encoder uses a BLIP-2-style Q-Former to bridge audio representations into the language model. The second is a video pipeline using MediaPipe FaceLandmarker to isolate and analyze oral region motion frame by frame, extracting articulatory features like tongue tip position, mouth opening geometry, and lip protrusion in real time from a standard webcam. The third component is SpeakMK1LLM, which serves as the project’s core reasoning engine. To balance cutting-edge architectural research with deployment stability for the current phase, we have developed two iterations of this model: The Hybrid Prototype: A custom 70M parameter Mamba-attention hybrid model designed for high-efficiency sequence modeling. It was trained on a rigorous four-stage curriculum: general pretraining on Tinystories, domain adaptation on CHILDES, clinical knowledge injection from pubmed central, and final instruction tuning. The Deployment Model: A fine-tuned Gemma 3n E4B, which leverages the same four-stage curriculum to adapt its large-scale reasoning capabilities to the specific nuances of Speech-Language Pathology (SLP). The entire stack requires only a webcam and a microphone, which is the point — making clinical-quality articulation feedback accessible without specialized hardware or a therapist present in the room. For the demo I would show the live pipeline taking a child's speech, flagging a specific phoneme error, and generating a graded corrective prompt in real time, alongside the clinician dashboard that logs session data for remote review.

- Event context: AI Tinkerers Dubai - May Demo Day — 2026-05-23 — Dubai
- Public talk page: https://dubai.aitinkerers.org/talks/rsvp_f9s0FLsNTT0

### [YT shorts finder](https://austin.aitinkerers.org/talks/rsvp_TpbSnDWuX0s)

A way to extract meaning from youtube shorts and search over your personalized database of them, using a mix of vision models, speech transcription models, and general purpose LLMs

- Event context: Community AI Demos - September Edition — 2024-09-12 — Austin
- Public talk page: https://austin.aitinkerers.org/talks/rsvp_TpbSnDWuX0s

## Related Technologies

- [Amazon Transcribe](https://aitinkerers.org/technologies/amazon-transcribe) ([Markdown](https://aitinkerers.org/technologies/amazon-transcribe.md)) — 2 public demos
- [BERT](https://aitinkerers.org/technologies/bert) ([Markdown](https://aitinkerers.org/technologies/bert.md)) — 179 public demos
- [BLIP-2 Q-Former](https://aitinkerers.org/technologies/blip-2-q-former) ([Markdown](https://aitinkerers.org/technologies/blip-2-q-former.md)) — 1 public demo
- [BLOOM](https://aitinkerers.org/technologies/bloom) ([Markdown](https://aitinkerers.org/technologies/bloom.md)) — 115 public demos
- [CMU Sphinx](https://aitinkerers.org/technologies/cmu-sphinx) ([Markdown](https://aitinkerers.org/technologies/cmu-sphinx.md)) — 2 public demos
- [Database](https://aitinkerers.org/technologies/database) ([Markdown](https://aitinkerers.org/technologies/database.md)) — 8 public demos
- [DeepSpeech](https://aitinkerers.org/technologies/deepspeech) ([Markdown](https://aitinkerers.org/technologies/deepspeech.md)) — 1 public demo
- [Facial Landmark Detection](https://aitinkerers.org/technologies/facial-landmark-detection) ([Markdown](https://aitinkerers.org/technologies/facial-landmark-detection.md)) — 1 public demo
- [Gemma 3n E4B](https://aitinkerers.org/technologies/gemma-3n-e4b) ([Markdown](https://aitinkerers.org/technologies/gemma-3n-e4b.md)) — 1 public demo
- [Google Cloud Speech-to-Text](https://aitinkerers.org/technologies/google-cloud-speech-to-text) ([Markdown](https://aitinkerers.org/technologies/google-cloud-speech-to-text.md)) — 2 public demos
- [GPT-3](https://aitinkerers.org/technologies/gpt-3) ([Markdown](https://aitinkerers.org/technologies/gpt-3.md)) — 191 public demos
- [GPT-4](https://aitinkerers.org/technologies/gpt-4) ([Markdown](https://aitinkerers.org/technologies/gpt-4.md)) — 529 public demos
- [IBM Watson Speech to Text](https://aitinkerers.org/technologies/ibm-watson-speech-to-text) ([Markdown](https://aitinkerers.org/technologies/ibm-watson-speech-to-text.md)) — 2 public demos
- [Kaldi](https://aitinkerers.org/technologies/kaldi) ([Markdown](https://aitinkerers.org/technologies/kaldi.md)) — 2 public demos
- [Llama-2](https://aitinkerers.org/technologies/llama-2) ([Markdown](https://aitinkerers.org/technologies/llama-2.md)) — 227 public demos
- [LoRA](https://aitinkerers.org/technologies/lora) ([Markdown](https://aitinkerers.org/technologies/lora.md)) — 17 public demos
- [Mamba-2](https://aitinkerers.org/technologies/mamba-2) ([Markdown](https://aitinkerers.org/technologies/mamba-2.md)) — 1 public demo
- [MediaPipe FaceLandmarker](https://aitinkerers.org/technologies/mediapipe-facelandmarker) ([Markdown](https://aitinkerers.org/technologies/mediapipe-facelandmarker.md)) — 1 public demo