Two disciplines quietly separate software that works on one laptop from software that survives a real firehose of data and a growing team. One moves data: event streaming and event-driven architecture. The other moves people: platform engineering and the internal tooling that lets a dozen engineers ship without each reinventing the same plumbing. They show up together on senior job specs for a reason, they are the load-bearing skills once a system gets big. Here is how each works, how teams actually use them, and exactly where to go to learn them.
Part 1: Event streaming and event-driven architecture
The core idea is simple and a little subversive: instead of services calling each other directly and waiting for an answer, they publish events to a durable log, and other services read from that log on their own schedule. The producer does not know or care who consumes the event. That decoupling is the whole game.
Apache Kafka: the durable log
Apache Kafka is the de facto backbone. Mentally, Kafka is an append-only log you can replay. Events are written to topics; each topic is split into partitions for parallelism and ordering; producers append, and consumer groups read independently, each tracking its own offset. Because the log is retained, a new consumer can start from the beginning and rebuild its view of the world, which is what makes patterns like event sourcing and change-data-capture possible.
What teams actually do with it: decouple a monolith into services that react to events instead of blocking calls; build real-time analytics and metrics pipelines; stream database changes (CDC) into search indexes and caches; and create an audit-friendly source of truth where the log is the history. The delivery and ordering semantics (at-least-once by default, exactly-once with the right configuration) are the details you have to get right, and the details that bite when you skip them.
Kafka Streams vs Apache Flink: processing the stream
A raw log is storage. The value comes from processing it, and that is where you choose a tool.
Kafka Streams is a Java library, not a separate cluster. You embed it in your own application, and it gives you stateful stream processing (joins, aggregations, windowing) that is Kafka-native and easy to deploy because there is no extra infrastructure. It is the pragmatic choice when your processing lives close to Kafka and you want to ship a service, not operate a platform.
Apache Flink is a full distributed stream processor. You run it as its own system, and in return you get serious stateful processing at scale: event-time semantics, watermarks to reason about late data, sophisticated windowing, and strong exactly-once guarantees with checkpointed state. When the processing is the product (complex event processing, large stateful jobs, sub-second analytics over huge volumes), Flink is the heavier, more powerful answer.
The concept that unlocks both is the difference between event time (when something happened) and processing time (when you saw it), plus watermarks and windowing to handle the messy reality that events arrive late and out of order. Get that mental model and the APIs stop feeling arbitrary.
Where to learn streaming
- Apache Kafka documentation and the Kafka Streams developer guide — start here, they are genuinely good.
- Apache Flink documentation — the DataStream API and the stateful-processing concepts pages.
- Kafka: The Definitive Guide, 2nd edition (2021) — the standard book; cite the 2nd edition for current material.
- Akidau, Chernyak & Lax, Streaming Systems — the conceptual bible for watermarks, windowing, and exactly-once. Read this and the rest clicks.
I get a small, honest version of these lessons running Corvus, the news aggregator behind my Apprised project. It is not Flink, but it is genuinely event-driven: scheduled workers pull and fold hundreds of feeds, every stage is idempotent so a retry never double-counts, late and duplicate items get reconciled, and a downstream consumer reads an exported corpus. Those three habits, idempotency, handling late data, and designing for replay, are exactly what Kafka and Flink formalize at scale. You can learn the instincts on a small system before you ever touch a cluster.
Part 2: Platform engineering and internal developer tooling
The second discipline is about people and friction. As an engineering org grows, every team re-solves the same problems: how do I stand up a service, wire logging, get a deploy pipeline, pass the security review? Platform engineering treats that repeated pain as a product, and the users are your own developers.
Golden paths and internal developer platforms
The central concept is the golden path (also called a paved path): an opinionated, well-supported, compliant-by-default way to do a common task, so the easy path is also the correct path. Instead of a wiki telling engineers to assemble fifteen tools, a golden path gives them a template that provisions a service with logging, health checks, a CI/CD pipeline, and the right security posture already wired in.
An internal developer platform (IDP) is the self-service layer that delivers those paths. The CNCF Platforms White Paper is the authoritative framing of what an IDP is and why it exists: to reduce developer cognitive load and let product teams ship without becoming infrastructure experts. The companion Platform Engineering Maturity Model is how you assess and grow the practice instead of cargo-culting it.
The other two pieces the job specs name:
- Shared libraries are the code-level version of golden paths: a common logging/auth/config/telemetry library so every service is observable and consistent by default, and a fix or upgrade propagates everywhere instead of being re-implemented per team.
- Developer experience (DX) is the measurable goal: shorter time-to-first-deploy, fewer steps to a new service, faster feedback loops, less toil. Good platform engineering is judged by how much friction it removes, not how clever the platform is.
How teams use it
In practice a platform team builds a software catalog and templates so "new service" is one click and is compliant from the first commit; standardizes the deploy pipeline so every team ships the same safe way; and curates the shared libraries that make logging, tracing, and metrics automatic. Backstage (created at Spotify, now a CNCF project) is the reference implementation of the catalog-plus-templates model, and a great way to see golden paths made concrete.
Where to learn platform engineering
- The CNCF Platforms White Paper and Platform Engineering Maturity Model — the conceptual foundation.
- Backstage docs — build a software catalog and a self-service template.
- The Kubernetes documentation — most platforms are Kubernetes-native, so the concepts and cluster-architecture pages are the substrate underneath.
How the two fit together
These are not separate worlds. On a mature team, the streaming backbone is itself a golden path: the platform team provides a blessed way to produce and consume events (a shared client library, a template that wires a service to Kafka with the right serialization, schemas, and observability), so an application engineer gets reliable event-driven plumbing without learning Kafka internals on day one. Streaming moves the data; the platform makes the data movement a one-click, compliant default. The same instinct shows up even at the edge: serverless primitives like Cloudflare Queues and Workflows are event-driven and durable-execution building blocks for the same patterns at a smaller scale.
Where to start
Pick the discipline you are weaker in and build the smallest real thing that exercises it. For streaming: a pipeline where producers push events to Kafka, a Kafka Streams or Flink job does a windowed aggregation with exactly-once, and results land in a sink, then deliberately feed it late and duplicate data and make it survive. For platform engineering: stand up a small cluster (k3s or kind) and build exactly one golden path, a template that provisions a service with logging, health checks, and a deploy pipeline baked in. A shipped system you can explain beats any certificate.
If you want the wider map these two disciplines sit inside, the AI/ML platform-engineering capability roadmap covers how streaming and platform work connect to model serving and agentic systems. The meta-skill under all of it is the same one behind every project here: build the smallest defensible thing, ship it, and let production teach you, which is the whole argument of The $97 Launch.
Related reading
- Becoming an AI/ML platform engineer — the full capability map these two disciplines belong to
- The Cloudflare developer platform — Queues and Workflows are event-driven primitives at the edge
- AI terminal kickstart — a working dev environment in one script
Fact-check notes and sources
- Kafka (log, topics, partitions, consumer groups, delivery semantics) and Kafka Streams (embedded library, stateful processing): Apache Kafka docs, Kafka Streams guide.
- Apache Flink (distributed stream processor, event-time, watermarks, exactly-once checkpointed state): Apache Flink docs.
- Books: Kafka: The Definitive Guide 2nd ed. (2021, Shapira, Palino, Sivaram, Petty); Akidau, Chernyak & Lax, Streaming Systems (2018).
- Platform engineering: CNCF Platforms White Paper and Platform Engineering Maturity Model; Backstage (CNCF, created at Spotify); Kubernetes docs.
This post is informational and reflects my own experience and reading; it is not professional advice, and I have no affiliation with the projects or products linked. Tool details and book editions are current as of mid-2026 and change over time.
