Project: tinygrad/tinygrad · Author: tiny corp / George Hotz and contributors · License: MIT
This is a reading note. I did not contribute to the project.

What it is

tinygrad sits between PyTorch and micrograd. Its goal is to be small enough to read: a compact Python codebase that still implements tensors, autograd, an IR compiler, JIT and multiple hardware backends — enough to train models and run LLM inference.

Core components

  • Tensor library: a PyTorch-like eager API with autograd
  • IR compiler: kernel fusion and lowering, JIT + graph execution
  • Training tools: nn, optim, datasets
  • LLM inference: GGUF support for mainstream open models

Hardware backends

CUDA / PTX, AMD ROCm, Apple Metal, OpenCL, Qualcomm QCOM, WebGPU, LLVM IR (CPU) and ONNX import.

The interesting part is that adding a new accelerator does not require rewriting the framework. You implement a small set of low-level ops, and the IR compiler handles fusion and scheduling.

Parts worth reading

Lazy execution and kernel fusion

With DEBUG=3 or 4, you can inspect fused kernels and generated code. JIT capture plus kernel replay cuts the overhead of many small ops.

It searches for the best kernel configuration per device. The underlying reality: the same graph can have very different optimal implementations on different GPUs.

Process replay

It compares generated kernels across changes to catch performance regressions — a step many “it runs” projects skip.

My take

tinygrad is best for two things: learning how a framework works internally, and validating a new hardware backend. It is not trying to replace PyTorch and is not a production training stack. But if you want to understand how an op becomes GPU code, it is more transparent than most alternatives.

View original project