Building Istio with Custom Envoy

1

I wrote a custom filter for envoy, and wish to build a version of Istio that allows me to use that custom filter. I tried following this blog post but ran into trouble on the make docker phase.

Looking in istio/bin/init.sh, it seems that if we set the variable USE_LOCAL_PROXY=1, then by default the build script should look for the envoy binary at ../proxy/bazel-bin/src/envoy/envoy. The only problem is that the init.sh script is run as an arg in a docker run command, where the image is specified by this dockerfile. Therefore, the filesystem on the container running the init.sh file does not have access to whatever code or binaries I put in the …/proxy directory on my machine.

I tried downloading the Dockerfile, and editing it to copy in my own envoy binary into the build container. I was able to build a copy of the build-tools container and do a hack on istio/common/scripts/run.sh to use my local image, but the problem is that my envoy binary was compiled for MacOS, not the OS that the istio build-tools docker container simulates.

I also tried unsuccessfully to copy the envoy code into the container and build it from scratch within the container (although I was able to do so on my own Mac box).

Now the only solution I can think of is the gross process of cross-compiling my envoy code for linux, editing the Dockerfile above to copy in the binary, hacking the run.sh file to use the resulting image, and then running make docker using the resulting setup.

I feel like I must be missing something blindingly obvious here; it can’t possibly be this difficult/clumsy to build a version of istio using a custom envoy binary. What gives?

Thank you for your help!

(Note—I was successfully able to compile my envoy binary using two methods: Just extending the envoy source code, for example see this github repo. The second way was by modifying the istio proxy repo and compiling it, resulting in the envoy binary at proxy/bazel-bin/src/envoy/envoy)

3 Likes
2

This is solved. I’m posting here in case anyone else is curious.

After much head-banging, I succeeded by:

  1. editing the istio proxy repo to include the envoy filter I wanted to add.*
  2. Building the repo inside the envoyproxy/envoy docker image.
  3. Copying the resulting envoy binary out, and dropping it into the /usr/local/bin/envoy of the istio proxyv2 image, found at this dockerfile: https://github.com/istio/istio/blob/085e01c8e4dfd165fce77af8df3e56882f69210b/pilot/docker/Dockerfile.proxyv2

To deploy the istio bookinfo app with your custom envoy:

  1. Turn off auto sidecar injection.
  2. Run istioctl kube-inject samples/bookinfo/platform/kube/bookinfo.yaml > out.yaml
  3. Edit out.yaml by replacing every occurrence of docker.io/istio/proxyv2:X.X.X with the name of the image you created in the previous section. Also, replace imagePullPolicy: Always to imagePullPolicy: IfNotPresent throughout the file.
  4. Run kubectl apply -f out.yaml

You should now have your filter running.

  • you need to specifically edit the istio-proxy repo because istio has written some custom filters without which istio doesn’t work.
1 Like
3

Hi @Colt_McNealy - This is great post. I’m new to Istio and would like to learn how to integrate custom envoy with Istio.

Do you mind sharing more details around the steps you took so I can reproduce it?

Thanks for you help.

1 Like