WEBVTT 00:00.000 --> 00:12.800 I'm Alex. I'm a software engineer for freedom of the presentation. I'm going to talk 00:12.800 --> 00:19.120 today about the subject of reproducible images. I want to do something before I even sign 00:19.120 --> 00:24.320 up a thing of the sort. How many people here have built irreproducible 00:24.320 --> 00:30.000 e-mots ever? Can I see a raise of hands? Yeah, and keep your hands up if you keep 00:30.000 --> 00:39.360 monitoring them ever since. Yeah, you're going to do it. Okay. So this talk will be about 00:39.360 --> 00:44.160 like why the rest of the people who didn't raise the hands should care about the subject 00:44.160 --> 00:50.560 of reproducible images. We're going to discuss how you can build one. Maybe we'll also 00:50.560 --> 00:57.680 offer some tools around that. And then we'll see, we'll go just a step beyond and we'll see 00:58.400 --> 01:07.280 what do we need besides building this image and reproducing it once? So I assume that if you're 01:07.280 --> 01:14.320 interested in reproducible builds, you have some sort of trust issues. I might trust issues 01:14.320 --> 01:19.680 are the fact that we're building an application in a freedom of the presentation. It's called 01:19.680 --> 01:28.480 Danger Zone. And this application is used to sanitize and trust documents with containers. 01:28.480 --> 01:32.480 This application is supposed to be used by an experienced users, journalists, activists, 01:32.480 --> 01:37.280 whistleblowers, and therefore it has to be auto updated. So we just point out later. There's 01:37.280 --> 01:43.040 no way to point out a specific digest. And both officially and unofficially, we kind of know 01:43.040 --> 01:51.280 that our application is used in within T-plines of major newsrooms. So you can understand that 01:51.280 --> 01:59.760 we are a bit frightened of what we serve to the users. What are our concerns? We continuously 01:59.760 --> 02:07.840 build our images in GitHub actions. GitHub is a US company. I'm not sure if we will trust 02:08.160 --> 02:13.760 this type of thing for activists and whistleblowers at this time and age. And even if we did, 02:14.960 --> 02:20.880 the rest of the users need to be able to verify. But what they get is what is what's 02:20.880 --> 02:27.760 currently built by the team of our main brands. So why we care that much about reproducible 02:27.760 --> 02:34.560 and why you make care as well? Because A people can audit the end result easily. Because people can 02:34.560 --> 02:41.440 know that the end result, people can also debug those can be detected very easily as well. 02:42.160 --> 02:49.920 And we do make some checks out of band too. And because those things happen, then my laptop, 02:49.920 --> 02:54.480 my hardware keys that I'm using to shine the images, the sci-fi plans that we're using, 02:54.480 --> 03:01.040 are not a viable target. So this gives us a piece of mind. That's why we have taken the last 03:01.040 --> 03:07.200 year to make our container image a bit for a bit reproducible. For those who don't know what 03:07.200 --> 03:13.040 we produce to build in means in this list with regards to container images, it's basically 03:13.040 --> 03:18.800 if you have like the build environment with all the files that you need to build the image, 03:18.800 --> 03:24.400 if you have a Docker file and if you have a build command, it should be able to create a bit 03:24.400 --> 03:29.840 for the same image if you run it now, five minutes later, one hour later, one year later. 03:32.080 --> 03:37.600 So let's see how we can create reproducible image. Basically, if you take a look at the 03:37.600 --> 03:45.360 documentation, you'll see that you need to set a source date epoch and pass some sort of 03:45.360 --> 03:52.400 rewrite timestamps flag, which we'll be used to set the timestamps of AIMAS layers to that epoch 03:52.720 --> 04:00.480 specified. And you can see here that this is a way Docker and behind the scenes build it and 04:00.480 --> 04:09.200 podman do this. Now, once you have, as you know, the commands you need to use, let's assume that 04:09.200 --> 04:14.000 you try to create an image that is reproducible. Is this similar to reproducible by the way? 04:14.080 --> 04:22.800 No way at all. And it's easy to see if you just rebuild it twice. But that's not the 04:22.800 --> 04:29.280 interesting thing. Most of you probably know why it's not reproducible. What's interesting is 04:29.280 --> 04:36.640 how can you assess, how can you understand why those two differences differ? What's the difference 04:36.720 --> 04:42.960 between those cases? You do see there's a difference, but why is that? And whenever I 04:44.000 --> 04:51.040 am faced with something like this, there is a very helpful project that offers some tooling around 04:52.000 --> 04:58.560 reproducibility and containers. It's reproducible containers project. I'm not affiliated with them. 04:59.040 --> 05:06.080 But it has been really cool to use. And I'm going to show you some tools from this project. 05:06.080 --> 05:14.160 First is the DFOCI tool, which you can pass it to images and you can also pass that you want to 05:14.160 --> 05:20.160 check just the file contents. And then you want to, for instance, dump just the differences in the file. 05:21.040 --> 05:28.320 And if you run this, for this example that I gave you, you will see that the differences are just 05:28.720 --> 05:35.680 some apt files. And we can further dig into that by, because we've done the contents, 05:37.280 --> 05:49.680 by doing the adiv of those files. And seeing that the difference here is that the 05:50.880 --> 05:57.840 somehow in the apt file and the log file, the host log has been captured and it. So the source 05:57.920 --> 06:04.880 date epoch that we used before is not a panache for everything. You need, you still need to do 06:04.880 --> 06:10.480 some stuff in order to make your image reproducible. And one may not even say, okay, let's just 06:10.480 --> 06:17.520 remove the logs. But that's not the full story, because we, where did we install this issue from? 06:17.520 --> 06:23.040 We sold it actually, even though it was a timestamp, there was a tag for this container image. 06:23.120 --> 06:27.040 We installed it from the main debutant repos. And this means that it can change at any time. 06:28.240 --> 06:32.960 So the solution, the solution to create a reproducible at least debutant image is to use 06:32.960 --> 06:38.640 the snapshot repos. Point them to something that aligns with your source date epoch. 06:39.440 --> 06:46.720 And then remove classes. And that's the second helper from this reproducible containers project that 06:47.600 --> 06:56.240 is very useful, which does this boilerplate stuff for us. And because you get stuff from the snapshot 06:56.240 --> 07:05.440 repos and you know that it's sometimes easier to to cache and reduce the loading snapshot 07:05.440 --> 07:14.960 service because they need the bandwidth. So on the left, we can see the original image we proposed, 07:15.040 --> 07:20.080 which is not reproducible. And here's the a bit more variable sting unfortunately, 07:20.080 --> 07:27.440 but this is the more reproducible thing on the right. And if you run this, the digest should always be 07:27.440 --> 07:40.240 that one. So we saw how we can build a reproducible image. But what can we do after that? 07:40.240 --> 07:46.640 And also, that image that I show you is like a two-line Dockerfile for production images. Most likely 07:46.640 --> 07:53.040 it's a much, much more complex image. And here are many more social from other terminism. 07:53.040 --> 08:00.320 We have been beaten when converting our own image from very stuff. We in order to make it fully 08:00.320 --> 08:06.400 reproducible, we have to build it and across different run science, we have to build it and across 08:06.400 --> 08:17.120 different days. And while I'm using the days thing, because while failing and debugging and 08:17.120 --> 08:23.200 failing and debugging and finding stuff, I came across like a pretty simple example to understand 08:23.200 --> 08:29.680 if like if the Dockerfile and sing can't be reproducible. And here's a pretty simple command. 08:29.680 --> 08:37.120 It's just lots of Dockerfiles have it. You are the user and in order to run in a less 08:37.120 --> 08:43.920 privileged user your application. So this, whichever Dockerfile has something like this, unless 08:43.920 --> 08:48.560 the person who wrote that really knows what they're doing, will be reproducible in five minutes 08:48.560 --> 08:55.360 from now. It will be one hour from now. It won't be in one day from now. And why is that? Because 08:55.520 --> 09:04.160 whenever you create a user in Etisado, the unique step of since you create a user is added there. 09:04.160 --> 09:12.640 So the reproducible variant is to do this add user chasing, at least that's what we found. 09:12.640 --> 09:18.800 I have seen very little examples of people doing that, which gives me the impression that 09:19.600 --> 09:26.080 there are very, very few images that are actually reproducible out there. So we have been 09:26.080 --> 09:34.720 it by so many things. We wanted some scientific checks, in order to feel that we are in the ground 09:34.720 --> 09:41.920 above us won't give in. So what we did is that we had this Dockerfile, they produced it all 09:42.000 --> 09:49.040 when we showed before. And we just created a CI job which will run a build command every night 09:49.760 --> 09:59.280 and will make sure that the image would always match the digest and at some point we were 09:59.280 --> 10:09.360 kind of bored with it because nothing failed. And then something did. And imagine our surprise, 10:09.360 --> 10:15.520 it is not supposed to happen. It's mostly a scientific check. And a couple of weeks in, 10:15.520 --> 10:24.320 you just stumbled on this thing. What was it? At that time when our CI test run built it, 10:24.320 --> 10:32.560 0200, had a regression, it added a small variant filled in the manifest. And it's a very 10:32.560 --> 10:37.440 very sequential change, but it changed a bit for a bit reproducibility that we strive for. 10:37.440 --> 10:45.120 We filed immediately a bug reports and the built-in maintainers fixed it very fast. And we thank 10:45.120 --> 10:50.720 you for that. But if we weren't reproducing this very image and this was out to the longer 10:50.720 --> 10:56.000 and people didn't notice, there will be images that could be reproducible with that particular 10:56.000 --> 11:03.040 version of build kit. And like once that bug was fixed, there wouldn't be able to be reproducible 11:03.040 --> 11:08.080 again. There would have to be the built version, there would have to be pins to that buggy, 11:08.080 --> 11:18.400 let's say, version. So we have collected lots of tooling and some workflows over the last year 11:18.400 --> 11:26.800 where we're doing this. And I wanted to show you here some, like, we have a repo, 11:26.800 --> 11:32.080 which you can do it, which you're going to say below, where with basically what we did is that 11:34.080 --> 11:40.400 we created scripts for all the incoming mistakes that we did over this last year. In order 11:40.400 --> 11:46.560 and to be able to do some stuff more easily, to verify images more easily, etc. So that people 11:46.720 --> 11:55.040 don't need to encounter them again and again again. And the main thing that this repo offers 11:55.040 --> 12:00.240 is a Python script. It doesn't have any dependencies, it pins build kit to a specific version 12:00.240 --> 12:06.720 solely to make sure that nothing changes and also compressive the latest we'll see that later. 12:06.720 --> 12:13.440 In a form some strict build parameters and it's supposed to be used both with Docker and 12:13.760 --> 12:19.520 Perman, CLI's and I'm using. I'm not using the cooler variants that are out there, 12:20.080 --> 12:25.680 because at the end of the day, what's the goal is not for us developers only to reproduce 12:25.680 --> 12:33.280 elements. The goal is to make it easier for any semi-technical person or a person who is not 12:33.280 --> 12:39.760 that well-versed in the container ecosystem to just have a way to reproduce an image. And those people 12:39.840 --> 12:47.440 may be Windows laptop and Mac OS laptop, etc. We also bundle this as a GitHub action and 12:48.480 --> 12:54.800 we have made it so that it's a drop-in replacement for the Docker build push-action, 12:54.800 --> 13:01.840 at least for simple builds, a drop-in as in the need to also define a source-day repo 13:01.840 --> 13:08.240 but I know that the interface is the same. And what's interesting and hopefully we'll be very helpful 13:08.320 --> 13:16.320 is that we also have a workflow, an action again, which does the following. It can build your image, 13:17.040 --> 13:25.520 it can build your project, an image from your project locally. And then you can pass it either 13:25.520 --> 13:30.240 a target image that you have push-summer, even the latest brands of that image so that you can 13:30.240 --> 13:36.960 check that the tip of your main branch matches to the latest sag of whatever register you're using. 13:37.040 --> 13:45.120 Or you can just paint it to a specific register and make sure that that's the exact same thing 13:45.120 --> 13:50.480 all the time. And that thing is something can easily run nightling. It should be a small change. 13:52.400 --> 13:58.240 We are also eating our own dog food in this repo and we are building nightling, 13:58.960 --> 14:06.800 bookroom and tricksy images and we offer them so that you can use them, 14:07.600 --> 14:13.600 using for testing purposes ideally because it's not super tested but the point is that you should 14:13.600 --> 14:19.920 be able to reproduce them independently. And we still reproduce this very little digest that 14:19.920 --> 14:26.160 helped us find the bug in the first place. Without said, there are still some stuff to do. 14:27.120 --> 14:34.480 While working on this, we realized that if people want to reproduce your image, 14:34.480 --> 14:39.920 they need to know what was the environment like a year after this image is out. They need to know 14:39.920 --> 14:46.320 what's the environment and they need to know what were the instructions you used in order to build 14:46.320 --> 14:50.800 that image. How many people will know that you use this repo build script? So ideally, 14:50.800 --> 14:58.160 it should be said described. What's also, what we also believe that is very critical in order to 14:58.160 --> 15:08.400 have a healthy ecosystem and be safer is to have something similar that Debian is doing and 15:08.400 --> 15:14.800 the rest of the reproducible build project, which have like some sort of registered packages in 15:14.800 --> 15:20.480 their case images could be in our case, where they are continuously monitored for aggressions. 15:22.880 --> 15:33.760 And we can see if there are the reproducibility presented false or increases. And finally, 15:33.760 --> 15:39.280 we would like some improved ergonomics for this thing, for multi-arch and complex build pipelines. 15:40.240 --> 15:51.280 So that's all for me. Thanks a lot for listening. And here's the links for the project, 15:51.280 --> 15:55.280 and the brief summary, what is discussed. Thank you. 16:04.320 --> 16:06.320 Questions? 16:10.240 --> 16:14.960 Hi. Thank you for the talk. This is exactly what there's a report. 16:14.960 --> 16:23.600 My question is, do you know, do you know, if there's a new report, the Docker file, if you want it to be 16:23.600 --> 16:30.640 reproducible, half of our file is now just this enabled and snap for repo, we're moving logs, etc. 16:31.520 --> 16:36.480 Is there, like, any base image that would have this old pre enabled or make it easier? 16:36.720 --> 16:39.760 And an image that has this pouch. 16:41.440 --> 16:44.640 An image that has those things, there's not such reports, etc. 16:44.640 --> 16:50.240 Back then, you said, right? Yeah, an image that would pre enabled the snap for repo, get to 16:50.240 --> 16:58.240 not log anything like we'd have installed, etc. Yeah, it doesn't make sense. That's the images 16:58.240 --> 17:05.120 that we are offering. The ones we see that we showed before pre-enable, 17:06.480 --> 17:10.560 the snaps at repo, they do not do something about logging, so that's something we need to 17:10.560 --> 17:17.600 remove on your own. But to be fair, for my experience, this is like just a tip of the iceberg. 17:18.320 --> 17:26.400 Even if this existed, there are so many things other than that that you need to figure out at some 17:26.400 --> 17:32.080 point because you're bound to hit into other limitations. So what I want to stress here is that 17:32.160 --> 17:37.280 it's good to have an image which has this reconfigured, but it's much more important to make it 17:37.280 --> 17:44.320 easy to do this build, check, beef, and then have this thing. So yeah. 17:48.720 --> 17:57.280 How slow is it for non-representable responses? What's the performance? How slow it is for the performance? 17:57.280 --> 18:10.480 Yeah, like non-representable. How fast they are. Because we rewrite the layers in order to set 18:10.480 --> 18:18.400 the source date Apple, this creates definitely an overhead. This creates an overhead when you 18:18.960 --> 18:24.800 rewrite the layers for the Apple. It's not as fast as regular images that for sure, but there are 18:24.800 --> 18:28.880 other reasons you're doing though, in theory. Thank you.