# Pre-training Projects at AI Tinkerers

> Canonical HTML: https://aitinkerers.org/technologies/pre-training
> Markdown URL: https://aitinkerers.org/technologies/pre-training.md
> Technology record last updated: 2026-02-26T20:13:40Z
> Generated: 2026-10-24T07:59:35Z

Pre-training is the foundational phase of deep learning: a model learns general features and world knowledge from massive, unlabeled datasets before being fine-tuned for a specific downstream task.

Pre-training is the critical first step in the transfer learning paradigm: the model (e.g., a Transformer architecture) is trained on a vast, general corpus—often petabytes of text like Common Crawl or Wikipedia—to establish foundational parameters. This initial phase uses self-supervised objectives, such as Masked Language Modeling (MLM) in BERT or next-token prediction in GPT models, to acquire a deep, generalized understanding of language or images. This 'base model' then only requires a second, efficient fine-tuning step on a smaller, labeled dataset to achieve state-of-the-art results (e.g., 90%+ accuracy) on specialized tasks, drastically reducing development time and computational cost.

- Official technology site: https://huggingface.co/models
- Public AI Tinkerers demos and talks: 1
- Result page: 1 of 1

## Recent Public Talks and Demos

### [Akkadian Oracle](https://la.aitinkerers.org/talks/rsvp_YvDWa0TWOiM)

- Event context: Jan 10 - AI Tinkerers Meetup - FOR BUILDERS — 2024-01-11 — Los Angeles
- Public talk page: https://la.aitinkerers.org/talks/rsvp_YvDWa0TWOiM

## Related Technologies

- [Akkadian Oracle](https://aitinkerers.org/technologies/akkadian-oracle) ([Markdown](https://aitinkerers.org/technologies/akkadian-oracle.md)) — 1 public demo
- [BERT](https://aitinkerers.org/technologies/bert) ([Markdown](https://aitinkerers.org/technologies/bert.md)) — 179 public demos
- [BLOOM](https://aitinkerers.org/technologies/bloom) ([Markdown](https://aitinkerers.org/technologies/bloom.md)) — 115 public demos
- [GPT-3](https://aitinkerers.org/technologies/gpt-3) ([Markdown](https://aitinkerers.org/technologies/gpt-3.md)) — 191 public demos
- [GPT-4](https://aitinkerers.org/technologies/gpt-4) ([Markdown](https://aitinkerers.org/technologies/gpt-4.md)) — 529 public demos
- [Llama-2](https://aitinkerers.org/technologies/llama-2) ([Markdown](https://aitinkerers.org/technologies/llama-2.md)) — 227 public demos
- [PaLM 2](https://aitinkerers.org/technologies/palm-2) ([Markdown](https://aitinkerers.org/technologies/palm-2.md)) — 116 public demos
- [RAG](https://aitinkerers.org/technologies/rag) ([Markdown](https://aitinkerers.org/technologies/rag.md)) — 148 public demos
- [RoBERTa](https://aitinkerers.org/technologies/roberta) ([Markdown](https://aitinkerers.org/technologies/roberta.md)) — 118 public demos