# Apache Spark Projects at AI Tinkerers

> Canonical HTML: https://aitinkerers.org/technologies/apache-spark
> Markdown URL: https://aitinkerers.org/technologies/apache-spark.md
> Technology record last updated: 2026-04-28T00:26:15Z
> Generated: 2026-09-23T10:38:20Z

Apache Spark is the unified, open-source analytics engine for high-speed, large-scale data processing across clusters.

Spark is your go-to engine for serious big data workloads: it's a unified platform designed for speed and versatility. The core advantage is in-memory processing, which makes it up to 100x faster than traditional MapReduce for iterative algorithms (like machine learning) and interactive queries. It provides high-level APIs in key languages—Scala, Java, Python (PySpark), and R—and integrates a full stack of tools: Spark SQL for structured data, MLlib for machine learning, Structured Streaming for real-time analytics, and GraphX for graph processing. Deploy it on Hadoop YARN, Kubernetes, or standalone; it handles batch, streaming, and advanced analytics on data of any size.

- Official technology site: https://spark.apache.org/
- Public AI Tinkerers demos and talks: 1
- Result page: 1 of 1

## Recent Public Talks and Demos

### [In-Context Learning for Extreme Multi-Label Classification](https://chicago.aitinkerers.org/talks/rsvp_y-7-uutZyio)

How to leverage large language models to identify > 10,000 different labels with < 10 labelled examples for each class? And how to do it by systematically searching the prompt space without tinkering with prompts for each class? There is this Stanford paper (From Chris Potts' group) and an accompanying Github repository that offers an answer to the questions above. I have been tinkering with it. The idea was to share what I am learning with fellow tinkerers in a quick talk.

- Event context: AI Tinkerers Chicago July Meetup — 2024-07-23 — Chicago
- Public talk page: https://chicago.aitinkerers.org/talks/rsvp_y-7-uutZyio

## Related Technologies

- [BERT](https://aitinkerers.org/technologies/bert) ([Markdown](https://aitinkerers.org/technologies/bert.md)) — 179 public demos
- [BLOOM](https://aitinkerers.org/technologies/bloom) ([Markdown](https://aitinkerers.org/technologies/bloom.md)) — 115 public demos
- [ECC](https://aitinkerers.org/technologies/ecc) ([Markdown](https://aitinkerers.org/technologies/ecc.md)) — 1 public demo
- [fastXML](https://aitinkerers.org/technologies/fastxml) ([Markdown](https://aitinkerers.org/technologies/fastxml.md)) — 1 public demo
- [GitHub](https://aitinkerers.org/technologies/github) ([Markdown](https://aitinkerers.org/technologies/github.md)) — 74 public demos
- [GPT-3](https://aitinkerers.org/technologies/gpt-3) ([Markdown](https://aitinkerers.org/technologies/gpt-3.md)) — 191 public demos
- [GPT-4](https://aitinkerers.org/technologies/gpt-4) ([Markdown](https://aitinkerers.org/technologies/gpt-4.md)) — 529 public demos
- [In-Context Learning](https://aitinkerers.org/technologies/in-context-learning) ([Markdown](https://aitinkerers.org/technologies/in-context-learning.md)) — 2 public demos
- [Llama-2](https://aitinkerers.org/technologies/llama-2) ([Markdown](https://aitinkerers.org/technologies/llama-2.md)) — 227 public demos
- [Mulan](https://aitinkerers.org/technologies/mulan) ([Markdown](https://aitinkerers.org/technologies/mulan.md)) — 1 public demo
- [PaLM 2](https://aitinkerers.org/technologies/palm-2) ([Markdown](https://aitinkerers.org/technologies/palm-2.md)) — 116 public demos
- [prompt space](https://aitinkerers.org/technologies/prompt-space) ([Markdown](https://aitinkerers.org/technologies/prompt-space.md)) — 1 public demo
- [RAkEL](https://aitinkerers.org/technologies/rakel) ([Markdown](https://aitinkerers.org/technologies/rakel.md)) — 1 public demo
- [RoBERTa](https://aitinkerers.org/technologies/roberta) ([Markdown](https://aitinkerers.org/technologies/roberta.md)) — 118 public demos
- [scikit-learn](https://aitinkerers.org/technologies/scikit-learn) ([Markdown](https://aitinkerers.org/technologies/scikit-learn.md)) — 84 public demos
- [TensorFlow](https://aitinkerers.org/technologies/tensorflow) ([Markdown](https://aitinkerers.org/technologies/tensorflow.md)) — 90 public demos
- [XML-CNN](https://aitinkerers.org/technologies/xml-cnn) ([Markdown](https://aitinkerers.org/technologies/xml-cnn.md)) — 1 public demo