LiteLLM 1.107.0 Ran When Python Started. Here Is the Check We Now Run Before Every Update

On October 21, 2026, someone published LiteLLM 1.107.0 to PyPI and pushed matching Docker images under LiteLLM's own names. It looked like an ordinary release. It carried three extra files that ran every time Python started on the machine, whether or not anything imported LiteLLM. Those files collected credentials, encrypted them and sent them out. A second piece sat in the background waiting for an address to download and run more code from.

LiteLLM is the model gateway on our AI server. Every local and hosted model call goes through it. So on October 24 I set out to answer two questions. Did the bad release ever run here? And what would have stopped it if it had arrived on a different day?

The first answer is no. The second answer is the uncomfortable one: our update command would have installed it. We pulled LiteLLM's main-stable image tag at 05:35 UTC on October 21, about six hours before the poisoned build went out under that tag, and again at 07:27 UTC on October 22, about seventeen hours after LiteLLM had pointed the tag back at a clean build. Timing kept us clean, not design. So we changed the design. This post covers what happened, how we checked, the indicator-of-compromise (IOC) questions worth asking on any small AI server, and the check that now runs before our server patches anything.

What happened on October 21

LiteLLM published its root cause on October 23, and it is worth reading in full. The short version starts five months earlier. On May 27, a configuration file committed to a public repository contained the master key to an internal alpha service, and that key was never revoked. On October 10 the attacker used it to pull a GitHub owner's command-line token out of the service, then pushed branches whose CI runs posted more secrets, including two other maintainers' tokens, to an outside endpoint.

On October 21 the owner's token turned off branch protection on the release repository and pointed the release job at a lookalike, Berri-AI/litellm, one hyphen away from the real BerriAI/litellm. A second maintainer's token approved the production gate. A third deleted the run records.

That is about an hour and 44 minutes on PyPI and about two and a half hours on Docker Hub, for a package Socket counts at roughly 23 million downloads a week. LiteLLM says every earlier version is safe and recommends pinning to 1.106.2 or another release published before the incident. Its private enterprise release was not affected.

Why starting Python was enough

Python reads every .pth file in its site-packages folder when the interpreter starts, and a line in one that begins with import runs as code. Python also imports a module named sitecustomize at startup if one exists. Both are documented features with legitimate uses.

The 1.107.0 wheels used both. They placed cook.pth, cook.py and sitecustomize.py in the wheel's data directory, so the files installed next to LiteLLM in site-packages rather than inside the litellm folder. cook.pth held one line, import cook. cook.py unpacked and ran a collector. sitecustomize.py started a detached loader that checked a file on GitHub every hour for a server address, then would download Python from that server and run it with no signature check. When researchers looked, the address file was empty, and nobody has reported a second stage being delivered.

Three consequences matter for anyone running AI tools:

• You did not have to use LiteLLM. Any Python start in that environment ran the collector: a test, a script, pip install of something unrelated. Microsoft reports a routine Python call that installed an unrelated dependency triggered it on Windows.
• Removing the package did not remove the payload. The three files lived beside the litellm folder, so deleting that folder left them in place.
• Looking from inside is risky. Running pip list or python -c in a suspect environment starts Python and runs the payload again. Microsoft and LiteLLM both say to inspect from outside, with tools that never start the suspect interpreter.

What it went after

Microsoft's and Socket's write-ups list the targets. Cloud credentials and instance metadata endpoints for AWS, Azure and Google Cloud. Kubernetes and Docker configuration. Git, npm and PyPI tokens, .netrc and .env files. SSH, GPG and TLS private keys. Browser logins, cookies and history, shell histories, and on Linux the readable environment variables of other processes. And AI tool sign-ins, by path: ~/.codex/auth.json, ~/.claude/.credentials.json, ~/.gemini/oauth_creds.json, ~/.cursor/mcp.json and others. If DATABASE_URL was set, it read up to 500 rows from LiteLLM's credentials and model tables, where a gateway keeps the upstream provider keys it exists to protect.

That target list describes an AI workstation. The machine that runs your model gateway is often the same machine where Claude Code, Codex and a browser are signed in. Ours is.

It was the second time this year

In March, LiteLLM 1.82.7 and 1.82.8 were published to PyPI with a stealer as part of the TeamPCP campaign, which had already hit Aqua Security's Trivy and Checkmarx's KICS GitHub Action. PyPI's advisory traced it to an API token exposed in the Trivy incident. Version 1.82.8 added litellm_init.pth, the same startup-hook trick. OSV's malicious-package record for the October release lists all three versions. Datadog notes that no connection between the two incidents had been established when it wrote its analysis.

Both releases came out of LiteLLM's own publishing path with real credentials, so a name check or a "verified publisher" badge would not have helped. Socket also notes that the October container build's provenance recorded the lookalike repository as its source. The provenance told the truth. Nothing downstream read it. A signature tells you who built something. It doesn't tell you whether the thing is safe.

How we checked our server

What runs, by digest. Our LiteLLM container runs version 1.106.2 from an image built at 09:31 UTC on October 21, before the malicious container build started. A Compose override pins it by its sha256 digest, so the tag cannot change under a running container. None of the three published bad digests is on the server.

Every copy, not only the one we meant to install. LiteLLM was on the server a second time, as a dependency of aider-chat (pinned by aider to 1.81.10) installed as a uv tool in the main user account. Nobody remembered that copy until the sweep listed it. Package caches count too: uv and pip caches, downloaded wheels, and every image layer.

Read files, never run the suspect interpreter. A read-only sweep hashed 148,965 cache, wheel and .pth files on the host and inside every container and compared them with the published hashes: zero matches. No cook.pth in any installed environment, no /tmp/.pc_once_* upload marker, no /tmp/remote_code_loader.lock, and none of the published marker strings. Images were read with docker create and docker export, which unpack the files without starting anything in them.

Ask the malware feed. OSV.dev's batch API answers with MAL-2026-2144 for litellm 1.107.0 and with nothing for 1.106.2.

Because the bad version never reached the server, we had nothing to rotate. That conclusion rests on all four checks together, not on a single missing marker file.

If it had reached us: how this setup would recover

We weren't hit, but I wanted the recovery written down while the details were fresh. How much work it takes depends on where the bad copy lands, and on this server it could have landed in two places.

Case 1: the gateway container

How it would have arrived. An update run between 11:46 and 14:20 UTC on October 21, following main-stable, would have pulled the poisoned image and restarted the gateway on it.

What the payload could have reached. I inspected the container to answer this rather than guess. It sees its own environment: the gateway's master key, the database URL with its password, the salt key that encrypts stored provider keys, and the admin login. It sees a read-only config file, two read-only folders of notes, and LiteLLM's database over a Docker network with no route out. With the salt key in hand, the provider keys stored in that database decrypt. The container also has outbound internet, which is how the upload would have left. It does not see the host's home directory, so the Claude Code and Codex sign-ins, SSH keys and GitHub tokens on the host were out of its reach. Its port listens on 127.0.0.1 only.

Recovery, in order:

1. Stop it. docker stop on the gateway ends every process inside it, including the loader. The stopped container's files stay in place, so export them for evidence before anything recreates it.
2. Roll back. update rollback on the gateway container recreates it from the digest pin saved before the update ran. Every run records each container's pin and start time first, and cleanup always keeps the rollback image. The payload read the database without writing to it, so the archived data doesn't need restoring.
3. Rotate from a clean device. Replace every provider key at the provider, because the stolen copy of the database holds the old ones. Then change the master key (LiteLLM's documented path is a new value in the environment and a restart), the virtual keys, the database password and the admin login. The salt key is different. LiteLLM's documentation says changing it makes the stored encrypted data unrecoverable, and there is no in-place migration. So plan it as a re-add: a new salt key, an emptied credentials table, then the models added again with their new provider keys. Read the two notes folders for anything secret and rotate that too.
4. Prove it before patching again. The update command refuses to patch while what runs matches an indicator, so update ioc-check has to come back clean and update health has to pass. Then watch each provider's usage page for calls you didn't make.

Case 2: the host copy

How it would have arrived. Not through aider itself. Aider pins its LiteLLM to an exact version, 1.81.10, so upgrading aider would not have picked up 1.107.0 by accident. The risk is any unpinned install in the main user account during those two hours: an agent job running pip install litellm for a quick script, or someone trying the gateway with uv tool install litellm. The forgotten aider copy shows how easily a LiteLLM lands in that account without anyone deciding to put it there.

What the payload could have reached. Everything that account can read: the Claude Code and Codex sign-ins, the GitHub, Netlify and Cloudflare command-line tokens, SSH keys, the Drive backup client's sign-in, and the backup repository's password file. On Linux it would also read the environment of every other process the account runs. That includes any key handed to a running job through an environment variable. This case is far larger than the first one.

Recovery, in order:

1. Don't start Python. Find the loader from outside with sudo fuser -v /tmp/remote_code_loader.lock, which names the process holding the lock, and kill it by process number.
2. Assume the upload succeeded and rotate everything. Work from a device that is neither this server nor the Windows VM running on it. Sign out every Claude and ChatGPT session. Revoke the GitHub tokens and keys. Re-run our GitHub Actions key setup, which issues a new forced-command key and replaces the repository secrets. Issue new Netlify and Cloudflare tokens and new tunnel credentials. Reauthorize the Drive backup client. Change the restic repository password: restic keeps several keys per repository, so add a new one and remove the old. Then replace the SSH keys and the model provider keys.
3. Rebuild, don't clean. The loader could have run anything an attacker served it while it lived. Nobody observed a second stage, but that can't be proved from inside the host. The rebuild path doesn't depend on SSH. Start from the provider's KVM console or rescue mode, reinstall from the setup guide, and reinstall the operations tooling from its GitHub repository with its installer. Containers come back from their digest pins. Restore data, not environments: databases and documents from the nightly restic snapshots and Drive copies, with tools and virtual environments rebuilt from lock files. A backup taken after the bad install carries the payload too.
4. Watch the accounts. Check the audit logs at GitHub, Cloudflare and Google, and the provider usage pages, for any use of the old credentials during the exposure window.

What made recovery possible, and what it showed us

Six things here would carry a recovery:

• run records that say to the minute which image ran when;
• digest pins with rollback images kept;
• nightly restic snapshots and Drive copies, plus a weekly restore test of the Qdrant and n8n backups into throwaway containers;
• the operations tooling in GitHub;
• console access that doesn't depend on SSH;
• a gateway container that can't see the host's sign-ins.

Writing the cases down also showed two gaps we haven't closed yet. The gateway container runs as root, keeps its default capabilities and has unrestricted outbound internet. An outbound allow-list naming only the model providers would have stopped the upload. And the forgotten aider copy lives in the account that holds every sign-in on the machine.

IOC questions for a small AI server

An indicator of compromise is any fact that lets you recognize the bad thing. The LiteLLM write-ups published several kinds, and each kind gets checked in a different place.

These are the questions I'd ask on any machine that runs AI tooling.

What does your tag mean right now? main-stable was one of the poisoned tags. Pinning by digest keeps a running container from changing. But an update that follows a floating tag moves the pin to whatever that tag names at that moment. That is exactly what our command did, twice, a few hours either side of the bad window.

Can you wait a day? The poisoned release was public for under two hours on PyPI. Socket flagged it in seven minutes. The OSV record that covers it was last modified at 22:01 UTC that night, about ten hours after the upload. A 24-hour hold on new releases would have skipped 1.107.0 entirely and given the feeds time to catch up. The cost is taking ordinary fixes a day late. For an urgent security fix the hold can be lifted for one run, and the other checks still apply.

What happens when the feed is down? Treat "could not check" as "do not install yet". A check that passes when it can't reach its data source isn't checking anything.

Is this startup hook new? Legitimate .pth files with import lines are common. On our host, virtual environments, setuptools' distutils-precedence.pth and namespace packages account for dozens. Telemetry libraries ship sitecustomize.py files too. Blocking every one would block normal software. Comparing the candidate with the version running now, and holding only hooks that are new until a person reads them, catches the October trick without stopping routine updates.

Where else is it installed? Check transitive installs, caches and image layers. Our second LiteLLM came in through a different tool.

What does a finding prove? Microsoft makes the point that package presence alone doesn't prove theft. Correlate hashes, process history, outbound traffic and later account activity. A missing upload marker doesn't prove safety either. LiteLLM's guidance is blunt: if 1.107.0 ran on a host with outbound internet, assume the theft succeeded. Rotate from a clean machine, as in the two cases above. Then revoke active sessions, because rotating a credential doesn't always end a session that already has a token.

Is every indicator malware? No. The same week, the CA eMudhra disclosed weak keys behind eight intermediate certificates, and Chrome and Mozilla removed trust in the emSign ECC Root CA - G3 root. Ubuntu's trust store on our host already distrusts it. Old copies of certifi's certificate bundle inside five of our container images still include it. We carry distrusted roots as warnings, not blocks, and the images get rebuilt.

Where did the chain start? It started with a key in a public repository that was never revoked. An AI workstation is full of keys: agent tokens, provider keys, deploy tokens. Keep them scoped to one job. Rotate on any exposure, even a brief one, and keep long-lived owner tokens off machines that install packages.

What we added to our update command

Our server has one command, update, that checks, patches, verifies and rolls back every part in a fixed order. The post about it covers the design. On October 24 we added a supply-chain gate. Nothing is patched in until the candidate passes it.

Run against October 21, the gate would have stopped 1.107.0 inside the window, before any list knew about it. Two holds would have applied. The image was minutes old. And it carried two startup hooks at the site-packages root that the running version didn't have. After 22:01 UTC that night, the OSV record would have blocked it outright.

A block fails that step, sends an alert and removes the pulled image, so a manual restart can't pick it up from the floating tag. The current version keeps running. A hold also leaves the current version running and lists the reason in the run report. If anything running now matches an indicator, the command refuses to patch at all until someone investigates. The npm dependency tree is resolved with install scripts turned off, before npm install runs anything.

The IOC list is a JSON file in the operations repository. Pushing a new indicator to the main branch applies it on the next run without a reinstall, and the merge can only add indicators, never remove them. The GitHub Actions workflow that can start our updates now runs the IOC check as its own step before any update, and stops if what runs now matches.

Tonight's check shows the gate doing its job on a clean release. LiteLLM's main-stable now carries 1.106.3, built at 06:46 UTC on October 24. The gate holds it for age until 06:46 UTC on October 25. Nothing is wrong with it. It's just younger than a day. The same check also lists what is ready:

  NEEDED  Host CLIs                      codex 0.163.0 -> 0.164.0, claude 2.1.308 -> 2.1.310, ...
  NEEDED  Service images                 LiteLLM, Ollama, n8n newer
  WARN    Supply-chain IOC check         running images clear (2 indicators ...; 3698 checked against the IOC list and OSV.dev)
            - WARN enclave-litellm-1: distrusted CA root 'emSign ECC Root CA - G3' still in .../certifi/cacert.pem
  ok      Health                         27 checks: 0 failed, 0 warnings

The Safe Update Command Kit now has a generalized version of the gate, and the operator kit has the checklist.

A check you can run tonight

These commands only read files and ask a public API. None of them starts Python in an environment you are checking. Run them on the host as a user who can read the folders involved.

Ask OSV.dev about a package version (it returns MAL- ids for known malicious releases):

curl -s https://api.osv.dev/v1/querybatch -H 'Content-Type: application/json' \
  -d '{"queries":[{"package":{"ecosystem":"PyPI","name":"litellm"},"version":"1.107.0"}]}'

List startup hooks with import lines and any sitecustomize modules on the host, skipping the container stores:

sudo find / -xdev \( -path /var/lib/containerd -o -path /var/lib/docker \) -prune -o \
  -path '*-packages/*.pth' -exec grep -l '^import' {} + 2>/dev/null
sudo find / -xdev \( -path /var/lib/containerd -o -path /var/lib/docker \) -prune -o \
  \( -name cook.pth -o -name cook.py -o -name sitecustomize.py -o -name usercustomize.py \) -print 2>/dev/null

Expect a few legitimate results, such as _virtualenv.pth and distutils-precedence.pth. What matters is a hook you can't explain, or one that is new since your last check. Hash anything unexpected with sha256sum and compare it with the published IOC hashes.

Look for the marker files on the host and in each running container (this runs ls, not Python):

ls -la /tmp/.pc_once_* /tmp/remote_code_loader.lock 2>/dev/null
for c in $(docker ps -q); do docker exec "$c" ls -la /tmp/remote_code_loader.lock 2>/dev/null && echo "marker in $c"; done

List the startup hooks inside an image without starting it:

cid=$(docker create IMAGE) && docker export "$cid" | tar -t \
  | grep -E '(^|/)(cook\.pth|cook\.py|sitecustomize\.py|usercustomize\.py)$|-packages/[^/]+\.pth$'
docker rm "$cid"

Find stray copies of a bad version in caches and environments:

find / -xdev -type d -name 'litellm-1.107.0*' 2>/dev/null

One caution from our own run: the first time I ran a host-wide search without skipping the container stores, it found two cook.pth files. Both were our own test fixture. The IOC gate's test suite plants a harmless one-line copy, next to a fake litellm-1.107.0.dist-info and with no cook.py, to prove the gate blocks it, and the build cache keeps the layers. Check images through docker export, and identify a match before you act on it.

What this does not catch

• Patient attacks. A hold of a day does nothing against an attacker who spends two years earning maintainer rights, as the XZ backdoor showed in 2024.
• Anything nobody has reported yet. An IOC list and a malware feed only know about what has been found. The new-hook hold and the age hold are there for the gap before a report.
• Signed operating system packages. OSV has no malicious-package feed for Ubuntu's archive, which Ubuntu signs. We check apt candidates against our own list only.
• What you install by hand. The gate guards the update command. A pip install typed into a terminal goes around it, so keep agent jobs and package installs away from the account that holds your sign-ins.

Related reading

• Updated on Disk, Old in Memory: the update command this gate now sits inside, with the Safe Update Command Kit.
• Nation-State Malware, Protestware, and the Packages Your AI Pulls In: the wider pattern of maintainer takeovers, worms and slopsquatting.
• Sovereign AI Workstation Best Practices: separating agent jobs from administrator access, and the full playbook.
• The whole server, from scratch: where LiteLLM sits among the other layers.
• Sovereign AI Made Simple: the series summary and every download.

Fact-check notes and sources

All read October 24 and 25, 2026 (UTC).

• Root cause and timeline: LiteLLM, attack chain record, published October 23, 2026, which the project calls its canonical source: the May 27 key commit, the October 10 and 12 token theft, the release override and approval, the 11:39:51 to 11:40:13 wheel uploads, the 11:46:56 image publication, quarantine by 13:23 and Docker Hub cleanup at 14:20.
• Exposure windows, safe versions and remediation: LiteLLM, security incident post: PyPI 11:39:51 to 13:23 UTC; GHCR unavailable by 13:51; Docker Hub 11:46 to 14:20; 1.106.2 recommended; enterprise release not affected; delete the three files and their caches, stop the loader holding the lock file, rebuild, assume theft if it ran with outbound internet, rotate the listed secrets.
• Payload, targets and detection: Microsoft Security, Inside the LiteLLM 1.107.0 supply chain compromise, October 23, 2026: data-directory placement, the collector, the loader, the 500-row database query, the Windows trigger, inspect from a clean system, correlate before concluding.
• Build timeline, scanner timing, provenance and AI tool paths: Socket, LiteLLM compromise: alerts about seven minutes after publication, roughly 23 million weekly downloads, provenance recording Berri-AI, the AI credential file paths, the empty address file.
• Second analysis: Datadog Security Labs, LiteLLM compromised on PyPI and Docker: the three-engineer token use, the deletion of the litellm folder not removing the files, no established link to the March incident.
• Malicious-package record: OSV.dev, MAL-2026-2144, modified 2026-10-21T22:01:01Z, listing 1.82.7, 1.82.8 and 1.107.0. The batch API answered MAL-2026-2144 for 1.107.0 and nothing for 1.106.2 at about 00:30 UTC on October 25, and PyPI's JSON API returned 404 for 1.107.0.
• March incident: Wiz, TeamPCP trojanizes LiteLLM: versions 1.82.7 and 1.82.8 on March 24, 2026, the Trivy token root cause per PyPI's advisory, litellm_init.pth in 1.82.8.
• Gateway keys: LiteLLM, master key rotation (new master key in the environment, then restart; leave the salt key alone) and security and encryption FAQ (stored credentials are encrypted with the salt key; changing it makes them unrecoverable).
• Backup keys: restic, Encryption and key management (restic key add, restic key remove).
• Recovery cases: what the gateway container can see comes from inspecting it on October 25, 2026 (environment variable names, mounts and networks; no values were printed). The backup schedule comes from the server's timers the same night. The two cases describe what we would do; neither was exercised.
• Startup hooks: Python documentation, site module: executable import lines in .pth files and the sitecustomize import.
• eMudhra: Qualys, identifying compromised eMudhra certificates: eight intermediates with weak keys, reported October 17, Chrome and Mozilla trust removal; Mozilla bug 2083449.
• Our server: our own update run records (LiteLLM pulls at 05:35 UTC October 21 and 07:27 UTC October 22), the October 24 read-only sweep, the October 24 gate install, and the update check run at 00:21 UTC October 25. Host counts of legitimate startup hooks come from the same night.

This post is informational, not security-consulting or incident-response advice. If you ran LiteLLM 1.107.0, follow LiteLLM's and your security team's guidance. Mentions of LiteLLM, BerriAI, PyPI, Docker, GitHub, Microsoft, Socket, Datadog, Wiz, Qualys, OSV, Ubuntu, Mozilla, Google Chrome, Anthropic and OpenAI are nominative fair use. No affiliation is implied. The hero illustration was generated with Ideogram from a generic brief.