TikTok's private mobile API, taken apart.
TikTok's Android app talks to a private HTTP API that is faster than the web endpoints and returns far more. I reverse-engineered it and now run it at billions of profiles and videos a day. The write-ups below explain how it works and where I got it wrong. The datasets are what came out.
Scraping TikTok's Mobile API
Four unrelated things have to be right before TikTok answers you, and getting any one of them wrong returns a clean HTTP 200 with an empty body and no hint which. Device registration and activation, the X-Argus cipher stack, regional hosts, TLS fingerprints, and the 24 endpoints on the other side.
Read →Tracking Half a Billion TikTok Sounds
Every sound on TikTok, counted once a day, on one machine, for 76 GB a year. Devices turn out to be a consumable with a request budget rather than something that recovers from a rate limit, fourteen of my confident conclusions turned out to be measurement artifacts, and the full column reference is at the end.
Read →Keeping ClickHouse and Elasticsearch in Sync
ClickHouse keeps every scraped row and Elasticsearch answers the page in 20 ms. In between: four watermark designs that disagree with each other, a prune job that came close to emptying an index, and two days spent duplicating an entire index every single night.
Read →