Series: Ansible Like a Pro

  • Part 1: Why Ansible Won Config Management — Config management landscape, why Ansible won, the architecture of ansible-navigator and execution environments
  • Part 2 ← You are here — Building a real execution environment: pinned and private collections, Python dependency conflicts, the additional_build_steps escape hatch, and a local registry for sharing
  • Part 3: Writing Your First Playbook with Navigator — Inventories, your first real playbook, the navigator TUI versus stdout, artifacts and replay, and the first-week mistakes

In Part 1 we built an execution environment. Twenty lines of YAML, one ansible-builder build, and we were running playbooks inside a container. That was the toy.

This is the real one. The kind you actually hand to a team.

The difference between the toy and the real thing is everything Part 1 skipped: collections pinned to versions that won’t break next Tuesday, a collection pulled from your own private Git repo, Python dependencies that don’t fight each other, the escape hatch for the stuff execution-environment.yml can’t express, and a way to share the finished image without paying Docker Hub or wrestling a corporate registry.

That’s this article.

First, lift the hood

Before we add complexity, look at what ansible-builder build actually does. There’s no magic in it — it writes a Containerfile and hands the file to podman. You can see that Containerfile without building anything:

ansible-builder create -f execution-environment.yml -c build-context/

create (not build) writes the Containerfile and its build context into build-context/ and stops. No image gets produced. Open build-context/Containerfile and you’ll find four stages:

  1. base — your parent image, plus pip and a Python interpreter.
  2. galaxy — ansible-galaxy collection install runs here, dropping collections into a build context for the next stage.
  3. builder — the clever stage. It introspects every collection you installed, reads each one’s requirements.txt and bindep metadata, compiles Python wheels, and resolves system packages.
  4. final — the image that ships. Collections, Python packages, and system packages get copied in from builder; the user, workdir, and entrypoint get set here.

Why care? Because every knob in execution-environment.yml is just a way to inject lines into one of those four stages. The prepend_* and append_* fields we’ll get to are literally RUN and COPY lines glued onto a stage. Once you’ve seen the Containerfile, the YAML stops being mysterious.

(There’s a sibling command, ansible-builder introspect, that does the same dependency readout builder’s middle stage does, exposed for collection authors. We’ll use it at the end to verify an image.)

The definition file, field by field

Here’s a grown-up execution-environment.yml. I’ll walk it.

---
version: 3

images:
  base_image:
    name: docker.io/redhat/ubi9:latest

dependencies:
  ansible_core:
    package_pip: ansible-core>=2.17,<2.18
  ansible_runner:
    package_pip: ansible-runner>=2.4

  galaxy: requirements.yml

  python:
    - kubernetes>=29.0,<30
    - PyYAML
    - jmespath

  system:
    - openssh-clients [platform:el9]
    - git-core [platform:el9]
    - sshpass [platform:el9]

options:
  package_manager_path: /usr/bin/dnf
  user: runner

version

Just 3. If you’re on ansible-builder 3.x and you omit it, builder assumes v1 and silently uses the old schema, and half your fields get ignored. Set it. Always.

images.base_image.name

The parent image. ansible-builder’s happy path is RPM-based — ubi9, fedora, RHEL. That’s what the docs test against and where you’ll have the fewest surprises.

Can you use Ubuntu, like Part 1 did? Yes, it builds — but you carry the weight yourself: point options.package_manager_path at apt-get and make sure a Python interpreter lands in the image. For anything you’d call production, start from ubi9 or fedora and save yourself an afternoon of debugging.

(No, there’s no build_image key in v3. There was one in v1 and v2. It’s gone. The builder stage derives from your base image now.)

dependencies.ansible_core / ansible_runner

Pin them. package_pip: ansible-core>=2.17,<2.18 is how you say “2.17.x, not 2.18.” Leaving these unpinned means “whatever pip grabs today,” which is exactly the problem execution environments were invented to solve. Don’t undercut your own tool.

dependencies.galaxy

Where collections come from. Three forms are allowed: an inline list, a multi-line string, or — my preference for anything real — a separate file:

dependencies:
  galaxy: requirements.yml

That file is where the interesting stuff lives, so it gets its own section below.

dependencies.python

Pure-Python packages your playbooks and collections need at runtime — kubernetes, boto3, netmiko. Inline list, or a requirements.txt filename. Inline specs follow PEP 508 (kubernetes>=29.0,<30), so use real version specifiers, not vibes.

dependencies.system

System packages, via bindep. This is how sshpass, git, and openssh-clients get into the image. bindep is platform-aware: [platform:el9] means “only on Enterprise Linux 9.” Use the platform tags — they’re what save you from trying to install an apt package into an RPM image the day you swap base images.

additional_build_steps

The escape hatch. It deserves its own section, further down.

options

package_manager_path (apt folks, this one’s you), and user — the account navigator runs as inside the container.

Collections: pin them, and pull your own

The requirements.yml is where Part 2 earns its keep. Two things matter in it: pinning, and private collections.

Pin everything

# requirements.yml
---
collections:
  - name: community.general
    version: "8.6.0"
  - name: ansible.utils
    version: "2.11.0"
  - name: kubernetes.core
    version: "3.0.1"

Every collection gets a version. This isn’t optional in any team setup I’d defend. Collections ship breaking changes between minor versions. Pin community.general to 8.6.0 and your EE behaves the same on every machine, forever. Leave it floating and you’ll eventually get a message from someone whose playbook started failing at 3am because 8.7 renamed a module.

This is the same lesson as Part 1’s cryptography story, applied one level up. The whole point of an EE is determinism. Don’t throw it away by leaving versions off.

Private collections from Git

Most teams have at least one collection that isn’t on galaxy — the one with the company’s internal modules. type: git handles those:

# requirements.yml
---
collections:
  - name: https://github.com/acme/internal-collection.git
    type: git
    version: v2.1.0          # a tag
  - name: https://github.com/acme/internal-collection.git
    type: git
    version: 9f3a2c1         # or a commit SHA — the safest pin there is

type: git turns the entry into a git clone. Pin with a tag, or — for real reproducibility — a commit SHA.

Now the part that trips everyone. The galaxy stage runs inside the container, not on your host. ansible-builder generates the RUN ansible-galaxy collection install line itself; it does not use your SSH agent and it cannot see your ~/.ssh. So if that git repo is private, running ssh-add on the host before you build does nothing — the clone runs inside a container that has no idea who you are.

The clean fix is to not ask the container for credentials in the first place. Pre-clone the collection on the build host, copy it into the build as a file, and point requirements.yml at the local copy:

# execution-environment.yml
additional_build_files:
  - src: ./internal-collection        # your pre-cloned checkout, living on the host
    dest: collections

# requirements.yml — the path is relative to the build context
collections:
  - name: ./_build/collections/internal-collection
    type: dir

type: dir installs from a local directory; type: file does the same for a built tarball (ansible-galaxy collection build makes one). No key, no token, no secret ever crosses into the image. The exact path under the build context shifts a little between builder versions, so run ansible-builder create from earlier and look in the context directory to confirm where additional_build_files actually dropped your file. This is the option I reach for first.

If you genuinely need a live pull instead of a pre-clone — a CI pipeline, say — resist the urge to drop a token straight into the URL. It works, and then the token lives in a file you’ll eventually commit. The proper alternatives (a deploy key, or a git http.extraHeader set from a build secret through prepend_galaxy) are real but fiddly, and they shift with your setup — I’m not going to hand you a recipe I haven’t run against your registry. Start with type: dir.

Heads up: never commit a requirements.yml with an embedded token. Git history is forever, and tokens get scraped within hours of a push.

Python dependencies, and the conflict you’ll hit

dependencies.python looks simple. It stops looking simple the day two collections ask for incompatible versions of the same package.

Here’s the scenario. kubernetes.core wants kubernetes>=28. Some networking collection you just added wants kubernetes<27, because its maintainer hasn’t touched it in a year. pip picks one, the other collection breaks at runtime, and the error message points at the wrong module. I have lost entire afternoons to exactly this.

What you’d want is pip constraints — a way to say “resolve everything, but keep kubernetes inside this range.” Here’s the honest part: ansible-builder v3 has no first-class constraints field. It’s a known gap, with an open feature request sitting in the tracker. So you get two practical options:

  1. Pin explicitly in dependencies.python. Put kubernetes>=29.0,<30.0 right in the list. It forces the resolver’s hand. Slightly ugly, but it works and it’s visible to anyone reading the file.

  2. Do a constrained install through additional_build_steps. Keep a constraints.txt, copy it in with additional_build_files, and:

additional_build_steps:
  append_final:
    - RUN pip3 install -r /tmp/requirements.txt -c /tmp/constraints.txt

This bypasses builder’s own dependency resolution. You get control; you lose the introspection-driven assembly. Reach for it only when option 1 can’t keep the peace between two stubborn collections.

The lesson is the same as with collections: pin, pin, pin. Floating Python versions inside a tool whose entire job is reproducibility is self-sabotage.

The escape hatch: additional_build_steps

Eight hooks, two per stage (base, galaxy, builder, final): prepend_* runs before the stage’s own instructions, append_* runs after. Each takes a list of Containerfile lines, or a multi-line string.

additional_build_steps:
  append_final:
    - ENV ANSIBLE_GATHER_TIMEOUT=30
    - RUN echo "acme-ee v1.0" > /etc/ee-build-info

That’s how you bolt on anything the YAML can’t express: an environment variable, a label, a compiled binary you COPY in. (Need to copy files in — a custom CA bundle, a config? That’s additional_build_files, which maps host paths into the build context; then you COPY them in an append_* step.)

Two rules, both learned the hard way:

  • Don’t put a USER directive here. It breaks the stage ordering. Set the runtime user with options.user.
  • Don’t reach for it first. If dependencies.python or dependencies.system can express what you need, use them. Those flow through builder’s introspection. additional_build_steps is raw Containerfile, which means it skips the dependency graph entirely. Use the front door. The escape hatch is for when there is no front door.

Build it, then share it (no cloud registry required)

You’ve got a real execution-environment.yml and a real requirements.yml. Build:

ansible-builder build \
  -f execution-environment.yml \
  -t acme-ee:1.0 \
  --container-runtime podman

That tags the image acme-ee:1.0 locally. Fine for you. Useless for your team. To share it without a Docker Hub account or a ticket to the corporate registry, run your own:

podman run -d \
  -p 5000:5000 \
  --restart=always \
  --name registry \
  docker.io/library/registry:2

Tell podman this registry is fine over plain HTTP — it’s local, and TLS on a loopback address would be theater:

# ~/.config/containers/registries.conf   (rootless)
# or /etc/containers/registries.conf     (system-wide)
[[registry]]
location = "localhost:5000"
insecure = true

Push the image up, then pull it from another machine:

# on the build host
podman tag acme-ee:1.0 localhost:5000/acme-ee:1.0
podman push localhost:5000/acme-ee:1.0

# on a teammate's machine — use the build host's IP, not localhost
podman pull 192.168.1.10:5000/acme-ee:1.0

That teammate needs the same [[registry]] ... insecure = true block in their registries.conf, pointing at 192.168.1.10:5000. Plain HTTP is the reason why.

There is no --push flag on ansible-builder build. I checked — it doesn’t exist. You build, then you push with podman. Two steps, on purpose.

Stop typing --eei every time

By now you’re running navigator with --eei localhost:5000/acme-ee:1.0 --pull-policy missing on every single command. Stop. Put it in a settings file:

# ansible-navigator.yml  (in your project root)
---
ansible-navigator:
  execution-environment:
    image: localhost:5000/acme-ee:1.0
    pull:
      policy: missing
    container-engine: podman

Now ansible-navigator run playbook.yml uses your EE automatically. Commit this file next to your playbooks and the whole team inherits the same default.

One gotcha worth knowing: pull.policy: missing fetches the image only if it isn’t already local. For a shared registry you usually want tag (re-pull if the tag moved) or always. never is for air-gapped boxes where the image was baked in ahead of time.

Prove it has what you think it has

Before you trust an EE, confirm it actually contains the collections and packages you asked for. Two ways in:

# what collections — and their Python/system demands — does a folder claim?
ansible-builder introspect ./collections

# what's actually installed inside the running EE?
ansible-navigator exec --eei localhost:5000/acme-ee:1.0 --mode stdout \
  -- ansible-doc -t collection -l

The first is for collection authors — it’s what builder’s middle stage runs internally. The second is the one you’ll use day to day: ansible-navigator exec runs a command inside the EE, and from there it’s just Ansible. ansible-doc -t collection -l lists every collection and module the image can see. For versions and install paths, ansible-galaxy collection list is the more thorough listing — run it the same way, inside the EE. If your private collection isn’t in that list, the problem is your requirements.yml, not your playbook.

What you’ve got now

By the end of this you should have:

  • A real execution-environment.yml with pinned ansible-core, pinned collections, and declared Python and system dependencies.
  • A requirements.yml that pulls a version-pinned public collection and a private one from Git.
  • An escape hatch (additional_build_steps) you reach for only when the YAML can’t express what you need.
  • The image pushed to a local registry, with a teammate pulling it.
  • Navigator pointed at it by default through ansible-navigator.yml.

This is an execution environment you can hand to a team without flinching.

Next in the series: Part 3: Writing Your First Playbook with Navigator — We stop building infrastructure and start using it. Inventories, the navigator TUI, stdout versus interactive mode, and the workflow mistakes that bite people in their first week.

This series assumes you read Part 1 and built its EE, or that you’re comfortable enough with containers for “build stage” to need no explanation. If pip version specifiers or bindep platform tags are new to you, slow down on the dependencies section — that’s where the real power and the real foot-guns both live.