The K8s Inner Loop is Driving You Crazy
Here’s the thing: Kubernetes is amazing. Orchestration at scale? Revolutionary. Declarative infrastructure? Chef’s kiss. But the moment you try to develop inside Kubernetes, something inside you dies a little.
You change a line of code. Run docker build. Push to your registry (wait, the registry is slow today). Update your Deployment manifest. Run kubectl apply -f deployment.yaml. Watch the pod spin up. Check the logs. Oh, the image didn’t pull right. Delete the pod. Wait for the new one. Check again. It’s been 90 seconds for a one-line fix. Your 2 AM self is furious.
The Kubernetes inner loop — build, push, deploy, wait, debug — is glacially slow out of the box. That’s the actual problem these tools solve: they collapse that loop. They watch your code, rebuild intelligently, sync binaries or whole filesystems into running pods, and skip the registry entirely. They save your sanity.
But here’s where it gets spicy: Garden, Tilt, and Skaffold each solve this problem with wildly different philosophies. One’s a minimalist. One’s opinionated. One’s trying to be your entire DevOps platform. Let’s figure out which one won’t make you hate Kubernetes development.
The Players
Skaffold: Google’s “Just Automate It”
Skaffold is Google’s answer to the K8s dev loop problem. It’s backed by actual cloud infrastructure money, battle-tested in enterprise shops, and philosophically, it’s the minimalist in this group.
The pitch: Write a skaffold.yaml file that says “watch these files, build with this tool (Docker, Buildpacks, Bazel, Jib), deploy with this deployer (kubectl, Helm, Kustomize), then run these tests.” Skaffold automates the whole pipeline in the mode you specify.
Modes matter:
- File sync mode (
sync:): Changes to source → Skaffold rebuilds the container image in your local cluster without pushing to a registry. Some files sync directly into the container without rebuild. - Build-push-deploy mode: Full Dockerfile build, push to registry, deploy. Slower, but works for any registry/cluster combo.
Flavor: No-bullshit, YAML-driven, plays nicely with your existing Helm charts and Kustomize overlays. You get exactly what you ask for, no surprises.
Tilt: The Opinionated Friend
Tilt is the engineer at your desk who has Opinions and is not shy about sharing them. It replaced Makefile in a lot of shops because it’s so good at the “what do I run next” problem.
The pitch: Instead of YAML, write a Tiltfile in a Python-like language (Starlark) that says exactly what you’re developing. Not just “deploy this,” but “when I change this file, rebuild this service, run these tests, and show me this custom resource in the sidebar.” Tilt gives you a beautiful local web UI that shows resource status, logs, and lets you trigger rebuilds by clicking buttons.
The feel: Opinionated UX, gorgeous UI, “things just work” if you follow the conventions. Feels more like a full IDE for Kubernetes than a build tool.
Garden: The Overachiever
Garden is the senior engineer who decided to solve Kubernetes dev and your entire CI/CD pipeline while we’re at it. It’s modular, it’s ambitious, and it will absolutely give you more features than you know what to do with.
The pitch: Define “modules” (deployables) and “actions” (tests, builds, deploys, tasks) in garden.yml. Every component is a module. Every workflow is an action. Want to run all tests before deploy? Action. Want to build this service, then run integration tests only if build succeeded? Action. Want to manage multiple environments (dev, staging, prod) in one config? Garden does that. Want to pull in another Garden project as a dependency? Yep.
The philosophy: Garden is trying to be a complete orchestration layer for both local dev and your CI/CD. It’s the overachiever of the group.
Skaffold in Action
Let me show you what Skaffold actually looks like:
apiVersion: skaffold/v4beta1kind: Configmetadata: name: my-service
build: artifacts: - image: my-registry/my-service docker: dockerfile: Dockerfile sync: manual: - src: "src/**" dest: /app/src
deploy: kubectl: manifests: - k8s/deployment.yaml - k8s/service.yaml
portForward: - resourceType: deployment resourceName: my-service port: 8000 localPort: 8000
test: - image: my-service structureTests: - ./test.yamlYou skaffold dev and it watches. Change src/main.py, Skaffold rebuilds the image, syncs the file into the container (the sync: block), and your code is live in seconds. No push, no pull, no pod deletion. It’s quick.
The honest take: Skaffold is boring in the best way. It does exactly what it says. If you already have Helm charts and Kustomize overlays, Skaffold plugs right in. It’s not trying to be your whole platform; it’s trying to be the missing piece between “code changes” and “code running in K8s.”
Tilt in Action
Here’s a Tiltfile:
load('ext://restart_process', 'docker_build_with_restart')
docker_build_with_restart( 'my-service', '.', entrypoint='python -m uvicorn main:app --reload', only=[ './src', ], live_update=[ sync('./src', '/app/src'), ],)
k8s_yaml('k8s/deployment.yaml')
k8s_resource('my-service', port_forwards='8000:8000')
local_resource( 'run_tests', 'pytest', deps=['src', 'tests'], trigger_mode=TRIGGER_MODE_MANUAL,)You tilt up and a local web UI opens on localhost:10350. You see your service status, logs, and a button to manually trigger tests. Change code, it reloads in the container instantly. Click the button, tests run. The UI shows failures in bright red. It’s polished.
The honest take: Tilt feels like a product, not a tool. Someone sat down and designed the experience. The Starlark language feels natural if you know Python. The UI is gorgeous. There’s a learning curve (Starlark has quirks), but once you’re in, you feel the difference. Great for teams that value developer experience and don’t mind the slight overhead of learning yet another config language.
Garden in Action
Garden config:
kind: Projectname: my-projectproviders: - name: local-kubernetes context: docker-desktop
modules: - name: my-service type: container build: dockerfile: Dockerfile services: - name: my-service ports: - name: http containerPort: 8000
actions: - kind: Build name: build-service module: my-service
- kind: Deploy name: deploy-service service: my-service
- kind: Run name: test-service image: my-service command: [pytest]
- kind: Run name: test-and-deploy dependencies: - build-service - test-service command: ["sh", "-c", "echo 'All tests passed'"]You run garden dev (or garden deploy, or build/test individually) and Garden orchestrates the whole thing. Actions depend on other actions. Modules reference other modules. You can ask Garden to “deploy everything that changed” or “run all tests before deploying.”
The honest take: Garden is powerful and flexible. If you’re already managing multiple services in a monorepo or polyrepo, and you want one tool to handle dev, test, and deploy, Garden’s your answer. The downside? It’s a lot of config. The upside? Once it’s wired up, it’s really wired up.
The Honest Comparison
| Feature | Skaffold | Tilt | Garden |
|---|---|---|---|
| Setup Time | 10 minutes | 30 minutes | 1-2 hours |
| File Sync Speed | Fast | Faster | Fast |
| Learning Curve | Gentle (YAML) | Moderate (Starlark) | Steep (modules/actions) |
| Multi-Service | Yes, via Helm/Kustomize | Yes, multiple load() | Native, first-class |
| Testing Integration | Yes (structureTests) | Yes (local_resource) | Yes (actions) |
| Team Collaboration | Great | Great | Excellent |
| CI/CD Expansion | Via Skaffold CI | Via Tilt plugins | Native CI/CD support |
| Beautiful UI | No | Yes | CLI-only |
| Debugging Support | Standard K8s logs | Beautiful log viewer | Standard K8s logs |
| Maturity | Stable (Google-backed) | Stable (Tilt.dev) | Stable (Garden.io) |
Real Talk: Pick Your Poison
Pick Skaffold if:
- You already have Helm charts or Kustomize overlays and want to keep using them.
- You’re a minimalist. You don’t want to learn yet another language.
- You’re bouncing between local dev and CI/CD and want the same config in both places.
- You’re solo or pair-programming and don’t need a fancy UI.
- You want something battle-tested in enterprises and don’t mind slightly slower inner-loop times.
Pick Tilt if:
- You care about developer experience and UI. You spend 8 hours a day in this loop.
- You’re managing 3-5 services locally and want one tool to orchestrate all of them.
- You like Starlark (it’s not hard, honestly, if you know Python).
- You have tests you run frequently and want a beautiful test-running interface.
- You’re on a team where “how do we run this locally?” is a question that comes up a lot.
Pick Garden if:
- You’re managing a monorepo or polyrepo with lots of interdependent services.
- You want the same tool for local dev, testing, and deployment.
- You’re willing to invest time upfront to avoid redoing config across dev/CI/deploy stages.
- You need sophisticated orchestration (e.g., “run build, then test, then integration tests, then deploy”) without writing shell scripts.
- You’re a mid-to-large team where consistency and repeatability matter.
The Dirty Secret
Honestly? All three are fine. They all solve the core problem: they make the K8s inner loop less painful. You won’t pick wrong. You might pick the one that’s wrong for you right now, but you can always switch.
I’ve seen teams run Skaffold at scale in enterprises. I’ve seen startups fall in love with Tilt’s UI and never look back. I’ve seen ambitious monorepos where Garden pays for itself immediately.
The real move is picking one, running <tool> dev for a week, and seeing if it makes you less angry at Kubernetes. If it does, you’re golden. If it doesn’t, grab the next one.
One More Thing
Here’s the thing nobody tells you: the Kubernetes inner loop doesn’t have to be slow. For too long, it just… was. Developers ate it. But these three tools proved that we can do better. Now you get to pick which flavor of “better” fits your brain.
Your 2 AM self will thank you for whichever one you choose.