Published on

Shipping conda environments to production using pixi

Authors

In the recent months, we have seen pixi become a popular package manager for the conda ecosystem. This is because it provides a simple and fast way to install environments and get started with coding. One huge advantage of pixi is that it comes with support for lockfiles out of the box. When using conda, we always need to deal with additional setup steps to ensure that our environments are reproducible. Pixi, on the other hand, provides a lockfile that we can commit to our repository. Also, when the lockfile changes, pixi makes sure that the environment on our machine is updated accordingly. For more reasons to use pixi over conda/mamba/micromamba, see the blog post from our friends at prefix-dev.

At QuantCo, we already migrated more than 40 repositories to pixi and are very happy with the results. This way, we can ensure that all developers are using the same environment and that our pipelines are reproducible.

However, with a new package management tool, we also need a new way to ship our environments to production. In this blog post, we show two ways to do that using pixi. First, we show how to install pixi environments in docker containers. Second, for situations where we don't have access to docker in our production environment (for example when the target environment is Windows), we show how to use our new pixi-pack tool to pack and unpack pixi environments.

Containerizing pixi environments

Uwe Korn has written a blog post on Deploying conda environments in docker containers. He explained how to create a Dockerfile using conda-lock and making it as efficient as possible. We will show how to do the same with pixi. As an example, we are using a simple FastAPI web server that we want to deploy to production. The code is available in pavelzw/pixi-docker-example.

Naive approach

We start out with the official pixi docker container, copy our code into our container, and then run pixi install --locked to install our environment.

FROM ghcr.io/prefix-dev/pixi:0.23.0 AS build

WORKDIR /app
COPY . .
RUN pixi install --locked
EXPOSE 8000
CMD [ "pixi", "run", "start" ]

Note: Please make sure that .pixi is in the .dockerignore file. Otherwise, docker will copy the environment into the container which could lead to problems when the build machine is a different platform or architecture than the target machine.

Let's look at the image size of this container:

$ docker image ls pixi-docker-example:1-naive-approach
REPOSITORY            TAG                IMAGE ID       CREATED         SIZE
pixi-docker-example   1-naive-approach   dff14d81f526   2 seconds ago   691MB

This is pretty big for a simple application that only displays Hello, World!. Looking at the layers of the container, we see that the pixi install step is the largest layer.

$ docker history pixi-docker-example:1-naive-approach
IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
dff14d81f526   2 seconds ago   CMD ["pixi" "run" "start"]                      0B        buildkit.dockerfile.v0
<missing>      2 seconds ago   EXPOSE map[8000/tcp:{}]                         0B        buildkit.dockerfile.v0
<missing>      2 seconds ago   RUN /bin/sh -c pixi install --locked # build…   577MB     buildkit.dockerfile.v0
<missing>      6 seconds ago   COPY . . # buildkit                             97.2kB    buildkit.dockerfile.v0
<missing>      20 hours ago    WORKDIR /app                                    0B        buildkit.dockerfile.v0
<missing>      8 days ago      COPY --chown=root:root --chmod=0555 /pixi /u…   36.7MB    buildkit.dockerfile.v0
<missing>      5 weeks ago     /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B
<missing>      5 weeks ago     /bin/sh -c #(nop) ADD file:a5d32dc2ab15ff0d7…   77.9MB
<missing>      5 weeks ago     /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B
<missing>      5 weeks ago     /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B
<missing>      5 weeks ago     /bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH     0B
<missing>      5 weeks ago     /bin/sh -c #(nop)  ARG RELEASE                  0B

Remove the cache directory

The problem with this approach is that pixi install not only installs the required packages but also places them in the rattler cache in ~/.cache/rattler. On our local machine, this is not a problem since we probably want to reuse the cache and the files are hardlinked, so they shouldn't take up additional space. In docker containers, on the other hand, the cache is not reused and the files are copied instead of hardlinked.

The simple fix for this is to remove the rattler cache directly after the pixi install (in the same step, so that it doesn't create a new layer).

FROM ghcr.io/prefix-dev/pixi:0.23.0

WORKDIR /app
COPY . .
RUN pixi install --locked && rm -rf ~/.cache/rattler
EXPOSE 8000
CMD [ "pixi", "run", "start" ]

We reduced our image size from 691MB to 402MB:

$ docker image ls pixi-docker-example:2-remove-cache
REPOSITORY            TAG              IMAGE ID       CREATED        SIZE
pixi-docker-example   2-remove-cache   ed7b1a5b39d2   1 second ago   402MB

$ docker history pixi-docker-example:2-remove-cache
IMAGE          CREATED         CREATED BY                                      SIZE      COMMENT
ed7b1a5b39d2   1 second ago    CMD ["pixi" "run" "start"]                      0B        buildkit.dockerfile.v0
<missing>      1 second ago    EXPOSE map[8000/tcp:{}]                         0B        buildkit.dockerfile.v0
<missing>      1 second ago    RUN /bin/sh -c pixi install --locked && rm -…   287MB     buildkit.dockerfile.v0
<missing>      6 seconds ago   COPY . . # buildkit                             97.1kB    buildkit.dockerfile.v0
<missing>      20 hours ago    WORKDIR /app                                    0B        buildkit.dockerfile.v0
<missing>      8 days ago      COPY --chown=root:root --chmod=0555 /pixi /u…   36.7MB    buildkit.dockerfile.v0
<missing>      5 weeks ago     /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B
<missing>      5 weeks ago     /bin/sh -c #(nop) ADD file:a5d32dc2ab15ff0d7…   77.9MB
<missing>      5 weeks ago     /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B
<missing>      5 weeks ago     /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B
<missing>      5 weeks ago     /bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH     0B
<missing>      5 weeks ago     /bin/sh -c #(nop)  ARG RELEASE                  0B

Multi-stage builds

In the docker history above, we can see a COPY --chown=root:root --chmod=0555 /pixi... layer that weighs in at 36.7MB. This is actually the layer from the ghcr.io/prefix-dev/pixi:0.23.0 image that installs pixi into the container. At runtime, we don't need the pixi binary, so we can use a multi-stage build to reduce the image size even further. Instead of running pixi run start, we can activate the environment using an activation script generated by pixi shell-hook and run the uvicorn command after activation.

Note: This is an alternative to the "remove the cache directory" approach. With multi-stage builds, we don't need to remove the cache directory, as it is not copied to the final image.

FROM ghcr.io/prefix-dev/pixi:0.23.0 AS build

WORKDIR /app
COPY . .
RUN pixi install --locked
RUN pixi shell-hook -s bash > /shell-hook
RUN echo "#!/bin/bash" > /app/entrypoint.sh
RUN cat /shell-hook >> /app/entrypoint.sh
RUN echo 'exec "$@"' >> /app/entrypoint.sh

FROM ubuntu:24.04 AS production
WORKDIR /app
COPY --from=build /app/.pixi/envs/default /app/.pixi/envs/default
COPY --from=build --chmod=0755 /app/entrypoint.sh /app/entrypoint.sh
COPY ./my_project /app/my_project

EXPOSE 8000
ENTRYPOINT [ "/app/entrypoint.sh" ]
CMD [ "uvicorn", "my_project:app", "--host", "0.0.0.0" ]

Please note that the path to the environment must be the same in both stages as conda environments are not relocatable. Using this trick, we can reduce the image from 402MB to 363MB.

$ docker image ls pixi-docker-example:3-multi-stage
REPOSITORY            TAG             IMAGE ID       CREATED         SIZE
pixi-docker-example   3-multi-stage   0c2add89dbdb   7 seconds ago   363MB

$ docker history pixi-docker-example:3-multi-stage
IMAGE          CREATED          CREATED BY                                      SIZE      COMMENT
0c2add89dbdb   7 seconds ago    CMD ["uvicorn" "my_project:app" "--host" "0.…   0B        buildkit.dockerfile.v0
<missing>      7 seconds ago    ENTRYPOINT ["/app/entrypoint.sh"]               0B        buildkit.dockerfile.v0
<missing>      7 seconds ago    EXPOSE map[8000/tcp:{}]                         0B        buildkit.dockerfile.v0
<missing>      7 seconds ago    COPY ./my_project /app/my_project # buildkit    532B      buildkit.dockerfile.v0
<missing>      7 seconds ago    COPY /app/entrypoint.sh /app/entrypoint.sh #…   617B      buildkit.dockerfile.v0
<missing>      49 minutes ago   COPY /app/.pixi/envs/default /app/.pixi/envs…   287MB     buildkit.dockerfile.v0
<missing>      49 minutes ago   WORKDIR /app                                    0B        buildkit.dockerfile.v0
<missing>      6 days ago       /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B
<missing>      6 days ago       /bin/sh -c #(nop) ADD file:7f5ee17de6aff2b67…   76.2MB
<missing>      6 days ago       /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B
<missing>      6 days ago       /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B
<missing>      6 days ago       /bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH     0B
<missing>      6 days ago       /bin/sh -c #(nop)  ARG RELEASE                  0B

Use a production pixi environment

One great feature that makes pixi stand out from all the other conda package managers is multi environment support. This means that we can define multiple environments with different dependencies in one pixi.toml file. For more information, please check out the official documentation.

For this, we can create a dev feature in our pixi.toml file where all our dependencies (like pytest, ruff, mypy, etc.) are installed. In addition to that, we create a default environment that contains all dependencies for development and a prod environment that only contains the dependencies needed for production. With solve-group we can make sure that all dependencies in the prod environment match exactly the dependencies in our default environment. This way, we can make sure that we test against the same dependencies that we deploy to production.

# ...
[dependencies]
fastapi = "*"
uvicorn = "*"

[feature.dev.dependencies]
pytest = "*"
ruff = "*"
mypy = "*"

[environments]
default = { features = ["dev"], solve-group = "prod" }
prod = { features = [], solve-group = "prod" }

FROM ghcr.io/prefix-dev/pixi:0.23.0 AS build

WORKDIR /app
COPY . .
RUN pixi install --locked -e prod
RUN pixi shell-hook -e prod -s bash > /shell-hook
RUN echo "#!/bin/bash" > /app/entrypoint.sh
RUN cat /shell-hook >> /app/entrypoint.sh
RUN echo 'exec "$@"' >> /app/entrypoint.sh

FROM ubuntu:24.04 AS production
WORKDIR /app
COPY --from=build /app/.pixi/envs/prod /app/.pixi/envs/prod
COPY --from=build --chmod=0755 /app/entrypoint.sh /app/entrypoint.sh
COPY ./my_project /app/my_project

EXPOSE 8000
ENTRYPOINT [ "/app/entrypoint.sh" ]
CMD [ "uvicorn", "my_project:app", "--host", "0.0.0.0" ]

By only installing the dependencies we actually need, we can reduce the image size from 363MB to 265MB.

$ docker image ls pixi-docker-example:4-prod-environment
REPOSITORY            TAG                  IMAGE ID       CREATED        SIZE
pixi-docker-example   4-prod-environment   730ff324486f   1 second ago   265MB

$ docker history pixi-docker-example:4-prod-environment
IMAGE          CREATED          CREATED BY                                      SIZE      COMMENT
730ff324486f   1 second ago     CMD ["uvicorn" "my_project:app" "--host" "0.…   0B        buildkit.dockerfile.v0
<missing>      1 second ago     ENTRYPOINT ["/app/entrypoint.sh"]               0B        buildkit.dockerfile.v0
<missing>      1 second ago     EXPOSE map[8000/tcp:{}]                         0B        buildkit.dockerfile.v0
<missing>      1 second ago     COPY ./my_project /app/my_project # buildkit    532B      buildkit.dockerfile.v0
<missing>      1 second ago     COPY /app/entrypoint.sh /app/entrypoint.sh #…   618B      buildkit.dockerfile.v0
<missing>      48 minutes ago   COPY /app/.pixi/envs/prod /app/.pixi/envs/pr…   188MB     buildkit.dockerfile.v0
<missing>      50 minutes ago   WORKDIR /app                                    0B        buildkit.dockerfile.v0
<missing>      6 days ago       /bin/sh -c #(nop)  CMD ["/bin/bash"]            0B
<missing>      6 days ago       /bin/sh -c #(nop) ADD file:7f5ee17de6aff2b67…   76.2MB
<missing>      6 days ago       /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B
<missing>      6 days ago       /bin/sh -c #(nop)  LABEL org.opencontainers.…   0B
<missing>      6 days ago       /bin/sh -c #(nop)  ARG LAUNCHPAD_BUILD_ARCH     0B
<missing>      6 days ago       /bin/sh -c #(nop)  ARG RELEASE                  0B

Use a distroless base image

Another significant portion of our image size is the base image: ubuntu:24.04 weighs in at 76.2MB. Instead of using a full-fledged Ubuntu image, we can use a distroless base image like gcr.io/distroless/base-debian12 provided by the distroless project which only contains the bare minimum to run a container. The only thing that is needed when running conda environments is a libc on our system.

Note: This is also the reason why we can't use alpine as a base image for conda environments, as alpine uses musl instead of glibc.

Since distroless images are barebones, they don't have a shell that can execute our entrypoint.sh. Instead, we need to look at the outputs of pixi shell-hook and set all env variables that are being set in the activation script accordingly.

# executed in build container, should match runtime environment
$ pixi shell-hook -e prod
export PATH="/app/.pixi/envs/prod/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
export CONDA_PREFIX="/app/.pixi/envs/prod"
export PIXI_ENVIRONMENT_NAME="prod"
export PIXI_PROJECT_ROOT="/app"
export PIXI_PROJECT_NAME="pixi-docker-example"
export PIXI_PROJECT_MANIFEST="/app/pixi.toml"
export CONDA_DEFAULT_ENV="pixi-docker-example:prod"
export PIXI_EXE="/usr/local/bin/pixi"
export PIXI_ENVIRONMENT_PLATFORMS="linux-aarch64,linux-64,osx-arm64,win-64"
export PIXI_PROJECT_VERSION="NO_VERSION_SPECIFIED"
export PIXI_IN_SHELL="1"
export PIXI_PROMPT="(pixi-docker-example:prod) "

Most environment variables are only needed for pixi itself, so we can remove them. The only variables that are needed are PATH and CONDA_PREFIX.

Note: Beware that this can differ from project to project as packages can define arbitrary activation scripts.

FROM ghcr.io/prefix-dev/pixi:0.23.0-bookworm AS build

WORKDIR /app
COPY . .
RUN pixi install --locked -e prod

FROM gcr.io/distroless/base-debian12 AS production
WORKDIR /app
COPY --from=build /app/.pixi/envs/prod /app/.pixi/envs/prod
COPY ./my_project /app/my_project
EXPOSE 8000
# from pixi shell-hook -e prod
ENV PATH=/app/.pixi/envs/prod/bin:$PATH
ENV CONDA_PREFIX=/app/.pixi/envs/prod
CMD [ "uvicorn", "my_project:app", "--host", "0.0.0.0" ]

This reduces the image size from 265MB to 209MB.

$ docker image ls pixi-docker-example:5-distroless
REPOSITORY            TAG            IMAGE ID       CREATED        SIZE
pixi-docker-example   5-distroless   b2b5c97f6f9c   18 hours ago   209MB

$ docker history pixi-docker-example:5-distroless
IMAGE          CREATED        CREATED BY                                      SIZE      COMMENT
b2b5c97f6f9c   18 hours ago   CMD ["uvicorn" "my_project:app" "--host" "0.…   0B        buildkit.dockerfile.v0
<missing>      18 hours ago   ENV CONDA_PREFIX=/app/.pixi/envs/default        0B        buildkit.dockerfile.v0
<missing>      18 hours ago   ENV PATH=/app/.pixi/envs/prod/bin:/usr/local…   0B        buildkit.dockerfile.v0
<missing>      18 hours ago   EXPOSE map[8000/tcp:{}]                         0B        buildkit.dockerfile.v0
<missing>      18 hours ago   COPY ./my_project /app/my_project # buildkit    532B      buildkit.dockerfile.v0
<missing>      18 hours ago   COPY /app/.pixi/envs/prod /app/.pixi/envs/pr…   188MB     buildkit.dockerfile.v0
<missing>      19 hours ago   WORKDIR /app                                    0B        buildkit.dockerfile.v0
<missing>      N/A                                                            5.89MB
<missing>      N/A                                                            12.8MB
<missing>      N/A                                                            233kB
<missing>      N/A                                                            346B
<missing>      N/A                                                            497B
<missing>      N/A                                                            0B
<missing>      N/A                                                            64B
<missing>      N/A                                                            0B
<missing>      N/A                                                            149B
<missing>      N/A                                                            0B
<missing>      N/A                                                            1.46MB
<missing>      N/A                                                            22.9kB
<missing>      N/A                                                            271kB

Note: The distroless images are not super handy to debug, as they don't have a shell. If we need to debug our container, we can rebuild it with the debug tag which includes a busybox shell.

FROM ghcr.io/prefix-dev/pixi:0.23.0-bookworm AS build

WORKDIR /app
COPY . .
RUN pixi install --locked -e prod

FROM gcr.io/distroless/base-debian12:debug AS production
WORKDIR /app
COPY --from=build /app/.pixi/envs/prod /app/.pixi/envs/prod
COPY ./my_project /app/my_project
EXPOSE 8000
# from pixi shell-hook -e prod
# add a /busybox to the PATH, otherwise we can't run any commands
ENV PATH=/app/.pixi/envs/prod/bin:$PATH
ENV CONDA_PREFIX=/app/.pixi/envs/default
ENTRYPOINT []
CMD [ "uvicorn", "my_project:app", "--host", "0.0.0.0" ]

We can spawn a shell in the container with the following command:

docker run --rm -it --entrypoint=sh pixi-docker-example:5-distroless-debug

Use pixi-pack to ship environments

In some scenarios we might not have access to docker but want to run stuff on the target machine directly. For this, we could install pixi on the target machine and run pixi install --locked -e prod and then start our application. However, in some scenarios, our production setup is sealed off from the internet and we can't install pixi or the packages we need.

There are two solutions to this problem, the first one is conda-pack and the second is conda-constructor.

conda-pack

conda-pack takes an existing conda environment, zips it up and ships it to the target machine. There, we can unzip it, activate the environment, then run conda-unpack to cleanup prefixes from the active environment.

conda-pack has the issue that if you tinker with your environment in some way, the packed environment might contain files that are not in a "clean" environment that you would get when installing everything from scratch. Thus, the environment.zip that we create at one point in time might be different from the environment.zip that we create at another point in time, even if all installed packages are the same. Also, conda-pack requires the conda environment to be installed on the build machine, which might not be wanted or possible in some scenarios. This is particularly annoying when cross-building environments for different platforms as conda doesn't allow us to install Windows environments on Linux and vice versa.

conda-constructor

conda-constructor is a tool that creates conda environments from a construct.yaml file.

name: my-environment
version: X
installer_type: all

channels:
  - conda-forge

specs:
  - fastapi
  - uvicorn

It reads the construct.yaml file, solves the environment, downloads the .conda and .tar.bz2 packages and puts them into a single binary file that can be shipped to the target machine. On the target machine, we can call this binary file and install it to the desired location. We could use pixi list --json and some bash magic to generate a construct.yaml file with explicit package pinnings and then use constructor to build an installer out of that. On Windows, the graphical installer works fine with a few kinks (constructor #813) but the command line installer has the issue that it doesn't provide any feedback on the progress or success of the installation (constructor #812). Also, the installer is designed to be a full-fledged installer that adds itself to the user's program list which is intended for some use cases but in our case, we just wanted to unpack the environment with no additional steps.

However, our use case was just a simple way to unpack a conda environment on the target machine without a fully fledged installer. For this, we created pixi-pack, a tool that takes a pixi environment and packs it into a compressed archive.

pixi-pack

pixi-pack demo

pixi-pack is a simple tool that takes a pixi environment and packs it into a compressed archive that can be shipped to the target machine.

It can be installed via

pixi global install pixi-pack

We can pack an environment with

pixi-pack pack --manifest-file pixi.toml --environment prod --platform linux-64

It looks at the solved dependencies for our environment from pixi.lock, then downloads all .conda and .tar.bz2 files and puts them into a single archive. The archive is a .tar file that contains all the packages as a local channel.

# environment.tar
| pixi-pack.json
| environment.yml
| channel
|    ├── noarch
|    |    ├── tzdata-2024a-h0c530f3_0.conda
|    |    ├── ...
|    |    └── repodata.json
|    └── linux-64
|         ├── ca-certificates-2024.2.2-hbcca054_0.conda
|         ├── ...
|         └── repodata.json

We can unpack the environment with

pixi-pack unpack environment.tar

which will create an env directory with the installed environment as well as an activate.sh (or activate.bat on Windows) script that can activate the environment. Since pixi-pack just downloads the .conda and .tar.bz2 files from the conda repositories, we can trivially create packs for different platforms.

pixi-pack pack --platform win-64

If we don't happen to have the pixi-pack binary on the target machine (no need to install it, we just need the executable as it's statically linked), we can also use conda or micromamba to install the environment.

tar -xvf environment.tar
micromamba create -p ./env --file environment.yml
# or
conda env create -p ./env --file environment.yml

Also, if we need to install additional files or scripts like the project code, we can create a conda package out of them using rattler-build and then add the package to the pack using pixi-pack --inject my-project.conda, see here for a simple example. This will add the my-project.conda package to the channel directory in the pack and ensure that it is installed next to the other packages when running pixi-pack unpack. Beware that this won't re-solve the environment, so we need to make sure that the package we are injecting is compatible with the other packages in the environment and all dependencies of our package are met in the pixi environment.

Conclusion

We've shown two ways to ship pixi environments to production. The first way - containerization - is the most common way to ship environments to production. It is easy to set up and maintain and can be used in most scenarios. The second way - using pixi-pack - is a fast, simple and reproducible alternative to conda-pack and conda-constructor that integrates well with the pixi toolchain and can be used in scenarios where we can't use docker or need a simple way to ship environments to production.