# Arctic Shift Starter Kit How to get at twenty years of Reddit conversation, what it costs you in disk and patience, and which of the three access routes to use for which job. Version 1.0, 2026-08-22. Every number here was measured on that date. Free to copy, fork and reuse. Attribution appreciated, not required. Source article: https://jwatte.com/blog/reddit-data-arctic-shift-for-business/ Project: https://github.com/ArthurHeitmann/arctic_shift --- ## 0. What this actually is Arctic Shift is one person's project to keep Reddit data available after the platform closed off the bulk access researchers had been using. It gives you three ways in, and they behave very differently. Picking the wrong one is the main reason people bounce off this. | Route | Cost | Latency | Reliable? | Good for | |---|---|---|---|---| | Web search UI | Free | Near-live | Yes | Looking something up once, checking if your own data is in there | | HTTP API | Free | About 34 seconds behind live | **Best effort** | Exploration, light monitoring, subreddit discovery | | Monthly dumps | Free, plus disk | Previous month | Yes, it is a file | Anything you need to depend on | The single most useful thing to understand before you start: **the API is a free service run by one person, and the endpoints that scan a lot of data fail under load.** Not sometimes in theory. I watched the same query succeed in 6.3 seconds and then fail four times in a row over the following twenty minutes. That is not a complaint about the project, which is generous and well built. It is a design constraint you have to plan around. **If a business process depends on it, use the dumps.** Use the API to work out what you want before you commit to downloading 80 GB. --- ## 1. Feasibility, in numbers you can plan against Measured 2026-08-22 from the project's own download links and the torrent listings. | What | Size | Realistic on | |---|---|---| | One recent month, everything | About 56 to 80 GB compressed | A laptop with an external drive | | July 2026 specifically | 80.27 GB, in two files of 23.21 GB and 57.06 GB | Same | | Everything, June 2005 to December 2025 | **3.80 TB** | A dedicated drive, and a weekend | | Top 40k subreddits only, 2005 to 2023-12 | Separate smaller torrent | A laptop | | Subreddit metadata, rules and wikis | Separate small torrent | Anything | Two things that follow from those numbers, and they are the whole planning question: **You almost certainly do not want the full archive.** A single month is enough to answer most business questions, and the top-40k-subreddits cut exists precisely so you do not have to take the long tail. Start with one month. **Work with the files compressed.** The project says this and it is right. Decompressed, these are several times larger, and there is no reason to unpack a file you are going to read once line by line. --- ## 2. The three routes, in the order you should try them ### Route one: the search interface, five minutes https://arctic-shift.photon-reddit.com/search Start here even if you intend to end up somewhere else. Type the thing you care about, see whether the conversation you imagine actually exists, and find out which subreddits it happens in. Most people discover at this stage that their assumption about where their customers talk was wrong, which saves them a download. This is also where you check whether your own username is in the dataset, which the project links prominently and which is the right thing for them to do. ### Route two: the API, for exploration Base URL: `https://arctic-shift.photon-reddit.com` Status: `https://status.arctic-shift.photon-reddit.com` What I measured on 2026-08-22: * **It is about 34 seconds behind live Reddit.** The newest post in a busy subreddit was 34 seconds old. This is not a historical archive with a delay. It is a near-live index. * **Cheap requests are effectively unmetered.** Forty-five requests in three seconds, all answered, no throttling response. There are `X-RateLimit-Reset` headers but I never tripped the limit with small queries. * **Expensive requests fail, and failure is the normal case for some shapes.** See section 3. ### Route three: the dumps, for anything real Download links: https://github.com/ArthurHeitmann/arctic_shift/blob/master/download_links.md They are torrents on Academic Torrents. Seed them afterwards, because this whole thing works because people do. Then: ```bash git clone --recursive https://github.com/ArthurHeitmann/arctic_shift.git cd arctic_shift pip install zstandard ``` You need Python 3.10 or later. Open `scripts/processFiles.py`, set `fileOrFolderPath`, put your logic in `processFile`, and run it. The script handles `.zst`, `.zst_blocks`, `.jsonl`/ndjson and `.json`. --- ## 3. The constraints that will surprise you All measured, all on 2026-08-22, all against the live API. **Keyword search must be scoped.** You cannot search the whole of Reddit for a word. Ask for a `body` search without a scope and the API tells you so directly: ``` 'body' query parameter requires one of: author, subreddit, link_id, parent_id ``` This is the single biggest shape constraint on the free API and it drives everything else. You do not go looking for who mentions your product anywhere. You pick the subreddits first, then search inside them. Section 1 of the industry playbook is about picking them well. **Aggregates work on narrow dimensions and fail on wide ones.** The project's own documented example, aggregating one author's comments by year, returned real counts in 8.8 seconds. Its other documented example, aggregating posts by author across a subreddit, failed. So did every subreddit-scoped aggregate I tried, including a seven-day window on a subreddit with under 150,000 subscribers. **The failure message is always the same and it is polite:** ```json {"data":null,"error":"Timeout. Maybe slow down a bit"} ``` Two different things wear that message. A slow failure, ten to fourteen seconds, is the query genuinely running out of time. A fast failure, a quarter of a second, is a cost guard declining to start. Treat the fast one as "this shape will never work" and the slow one as "maybe at a quieter hour". **What is reliably fast.** Subreddit metadata lookups came back in 377 milliseconds. Listing recent posts in a subreddit is quick. ID lookups take up to 500 ids in one call. Comment trees for a single thread accept a limit up to 25,000 and are scoped by definition, so they behave. The pattern: **anything scoped to one thing is fine, anything that scans is not.** --- ## 4. What is in the data, and what has been done to it Read this before you draw a conclusion from it. The project documents its own modifications, which is more than most datasets do. * All data comes through the official Reddit API. **Content from private or quarantined subreddits is not included.** If your question is about a private community, this dataset cannot answer it and no amount of querying will change that. * One JSON object is one post or one comment. Keys are sorted, text is UTF-8, objects are ordered by `created_utc` then `id`. * From the 2023-04 dumps onward, `body_html` is removed and `retrieved_on` is added. * From 2024-04 onward, archives are `.zst` only. ### The second retrieval, which is the analytically interesting part From the 2023-11 dumps onward, everything was fetched a second time after a 36 hour delay, and the two versions were merged. Scores and counts are updated to the later values, and a `_meta` field records what changed: | Field | Meaning | |---|---| | `_meta.retrieved_2nd_on` | Timestamp of the second fetch | | `_meta.was_deleted_later` | It existed at first fetch and was gone 36 hours later | | `_meta.was_initially_deleted` | It was deleted at first fetch and available later, and the text is restored | | `_meta.removal_type` | More detail on why something was removed | | `_meta.is_edited` | The text changed between the two fetches | | `_meta.note` | `"initially_unavailable"` where that applies | **This is a genuinely unusual research asset and an ethical problem at the same time.** It lets you study what gets deleted and what gets edited, which is a real question in moderation research and in understanding a community. It also means the archive knowingly retains text that a person chose to remove. Both of those are true at once, and you should decide which side of it you are on before you build something. The fields that update on the second pass, `score`, `ups`, `downs`, `num_comments`, `upvote_ratio` and others, are 36 hour values. If you are studying engagement, that is your measurement window, and it is not the same as a post's final score. --- ## 5. Before you build anything on this Four things to settle, none of them technical. **The repository has no license.** GitHub reports no license for the project. That is not a detail to skip past if you are putting this inside a product. No license means no granted permission, whatever the practical norms are. If your use is commercial, this is a question for a lawyer, not for a README. **The data came from Reddit, and Reddit has terms.** The project collected it through the official API. That does not automatically make your downstream use of it permitted, particularly if you are redistributing, training on it, or reselling analysis. Again: a question to answer deliberately. **People did not consent to your analysis.** Everything in here was written by a person who was talking to a community, not filling in your market research. That is legal to read and it should still shape what you do. Aggregate patterns are one thing. Building a file on a named individual because they posted about a health problem in 2019 is another, and the fact that it is technically possible is exactly why it needs a rule rather than a shrug. **Do not republish individuals.** Quote a community, not a person. If you must quote text, do not attach the username, and think hard before quoting anything from a subreddit about health, addiction, abuse, sexuality, immigration status or money trouble. The archive is not the place those words were written. The project maintains a removal request form and links removal forms for the other archives too. That is the correct posture and it is worth matching. --- ## 6. Your first hour * [ ] Search the web interface for the two or three terms your business turns on. Note which subreddits come back. * [ ] Look up those subreddits for subscriber counts and creation dates. This tells you whether the community is big enough and old enough to be worth anything. * [ ] Read fifty posts by hand. Actually read them. Every automated pass you write afterwards will be better for it, and you will usually find that the thing you assumed people complain about is not the thing they complain about. * [ ] Write down the single question you want answered, in one sentence, before you download anything. * [ ] If the question needs more than a scoped search can give you, download one month and run it locally. The step people skip is the third one, and it is the one that determines whether the rest is worth doing. --- ## 7. When it outgrows one person at a keyboard Once you are asking the same question of eight communities, this becomes a fan-out problem with a verification requirement, which is exactly the shape the agent-fleet operating model is for. There are four charters for it, specific to this dataset, in section 9 of `arctic-shift-query-cookbook.md`: a scout that finds communities, a harvester that collects text, an analyst that clusters it, and a verifier whose entire job is to disbelieve the analyst. That fourth seat is the one that matters. A model asked to summarise a pile of Reddit threads will produce a confident, well-written consensus whether or not one exists in the data. The verifier pulls the cited ids back out of the raw file and checks that the quoted text is really there. The general version of the model, including why the verification seat outranks the workers, is at https://jwatte.com/blog/claude-code-agent-fleet-org-chart/ --- Companion files at https://jwatte.com/downloads/ * `arctic-shift-industry-playbooks.md` : concrete plays for laundromats, car washes, self-storage, trades, and larger organisations * `arctic-shift-query-cookbook.md` : copy-paste queries and local processing recipes * `find-your-subreddits.mjs` : a small script that finds and sizes the communities your customers are in * `agent-fleet-playbook.md` : the general operating model the four research charters are an instance of Written by J.A. Watte. https://jwatte.com