| *** xjuan has quit IRC | 03:19 | |
| *** csoriano has quit IRC | 04:04 | |
| *** tristan has quit IRC | 06:05 | |
| *** tristan has joined #buildstream | 06:39 | |
| *** ChanServ sets mode: +o tristan | 06:46 | |
| *** csoriano has joined #buildstream | 09:15 | |
| * ironfoot fails to use bst shell | 09:48 | |
| ironfoot | currently doing: `bst shell -b /root/buildstream/build/usbutils-c8lvnccq elements/foundation/usbutils.bst` | 09:48 |
|---|---|---|
| tristan | ironfoot, yeah I was there too | 09:56 |
| tristan | ironfoot, I dont understand yet what went wrong, been looking at busybox / bash artifacts from baserock / buildstream | 09:57 |
| tristan | ironfoot, fwiw, /usr/bin/ldd exists, and it has this at the top: #! no | 09:57 |
| tristan | maybe I should check glibc artifact | 09:58 |
| tristan | ironfoot, however what I noticed is that when /bin/sh is a link to bash, usbutils passes | 09:58 |
| tristan | so I was wondering if perhaps in baserock at that stage, /bin/sh was bash instead of busybox | 09:58 |
| ironfoot | i believe that /bin/sh was always busybox | 10:00 |
| tristan | Ok | 10:00 |
| tristan | is there a /usr/bin/sh that is bash ? | 10:00 |
| tristan | because PATH has /usr/bin before /bin | 10:01 |
| ironfoot | nope | 10:01 |
| ironfoot | and /bin/sh is busybox | 10:01 |
| ironfoot | if I run manually the bootstrap, i also get "/bin/sh: ./bootstrap: not found" | 10:02 |
| tristan | aha | 10:02 |
| tristan | found it | 10:02 |
| tristan | ironfoot, not if you make /bin/sh become bash, though | 10:02 |
| tristan | I tested that... but now I found the bug :) | 10:03 |
| ironfoot | please, tell me :) | 10:03 |
| tristan | ironfoot, the problem is (as I suspected in build-essential, where I did needed to do some more hands on converting) with glibc | 10:03 |
| ironfoot | (`sh bootstrap` also "works") | 10:03 |
| tristan | ironfoot, so with baserock, /usr/bin/ldd gets generated with #! /usr/bin/bash | 10:03 |
| tristan | with buildstream, it's #! no | 10:04 |
| ironfoot | aha | 10:04 |
| tristan | something about detecting bash when building glibc | 10:04 |
| ironfoot | rings a bell | 10:04 |
| tristan | perhaps its fakebash I got wrong or smth | 10:04 |
| ironfoot | aaaaaaah | 10:04 |
| ironfoot | you need fake-bash :D | 10:04 |
| tristan | Do I not have it ? | 10:04 |
| ironfoot | let me have a look | 10:05 |
| tristan | nope | 10:05 |
| tristan | hah | 10:05 |
| ironfoot | it's there, but not included I guess | 10:05 |
| tristan | Aha ! | 10:06 |
| * ironfoot ponders adding a check after building glibc, to see if some scripts have a `no` in the first line | 10:08 | |
| tristan | ironfoot, I'm rebuilding the world | 10:08 |
| tristan | nah I dont think so | 10:08 |
| tristan | well, up to you but... | 10:08 |
| ironfoot | yeah, it will take a while, but that will fix some other broken bits | 10:09 |
| tristan | in any case, the conversion above build-essential being entirely automatic and with no commits is important | 10:09 |
| ironfoot | indeed | 10:09 |
| tristan | and nice to see it's going well :) | 10:09 |
| tristan | hehe | 10:09 |
| tristan | ironfoot, fwiw I did not encounter your findutils build error | 10:09 |
| tristan | it just built fine here | 10:09 |
| *** ssam2 has joined #buildstream | 10:09 | |
| ironfoot | tristan: it worked well when i tried again | 10:10 |
| ironfoot | and that is a bit scary | 10:10 |
| tristan | ironfoot, so its possible we have an 'incident' which might occur again | 10:10 |
| tristan | yeah | 10:10 |
| tristan | ironfoot, one thing I've been keeping a secret, is that the toolchain is not baserock in the triple after bootstrapping (instead it remains 'unknown') | 10:14 |
| tristan | I worked around that with the substitution of $TARGET with $(gcc -dumpmachine) | 10:15 |
| tristan | But | 10:15 |
| tristan | ironfoot, I maintain that whether we fix that problem or not, beyond the bootstrapping process itself $(gcc -dumpmachine) should be used *anyway* | 10:15 |
| tristan | this makes build instructions more flexible for being built on top of different bases | 10:16 |
| tristan | also, when we modularize the bst definitions, I dont even want that knowledge of %{target} to exist in depending projects | 10:17 |
| tristan | (core would not have %{target} which is defined in build-essential's project.conf) | 10:17 |
| ssam2 | seems fair, i never thought of using $(gcc -dumpmachine) | 10:19 |
| * ssam2 wonders if llvm/clang supports the same argument | 10:19 | |
| tristan | ssam2, I would think so | 10:22 |
| tristan | I mean, the compiler is the one who knows about the base runtime | 10:22 |
| ssam2 | sure, just wary of hardcoding calls to `gcc` in our definitions -- it means we then have to always use a compiler called `gcc` | 10:23 |
| ssam2 | but in practice we do anyway right now :-) | 10:23 |
| tristan | indeed, it could be changed to `cc` | 10:23 |
| ssam2 | good plan | 10:23 |
| tristan | ironfoot, moment of truth, stage3 glibc is building... | 10:24 |
| ssam2 | i guess odd compiler configs (like having to use /opt/foo/xlC) won't be thing with BuildStream because there's no reason to have 2 x C compilers in an SDK | 10:24 |
| ssam2 | *won't be a thing | 10:24 |
| tristan | ssam2, that doesnt really change much actually | 10:25 |
| tristan | if you want to bootstrap a new runtime, you need the sysroots | 10:25 |
| tristan | whether your host is your real host, or an imported SDK | 10:25 |
| ssam2 | sure, my worry is that assuming `cc` from PATH is always the C compiler you want to use... but anyway, this is miles in the future | 10:26 |
| tristan | maybe not so many miles :) | 10:26 |
| ssam2 | i can imagine some weird legacy components that only build with a weird legacy C compiler, but the rest of your components all build with normal GCC | 10:26 |
| tristan | ssam2, I'll test it with `cc` and if it works, we can convert definitions with that subst | 10:27 |
| ssam2 | it'll work for Baserock definitions for sure | 10:27 |
| ssam2 | anyway, I guess weird legacy components will need special case hacks in any case | 10:27 |
| tristan | Sure, in practice every build chooses it's dependencies | 10:28 |
| ssam2 | also true | 10:28 |
| tristan | that will be strengthened by modularization too, but I need to write the recursive pipeline element for that to be possible | 10:28 |
| tristan | ironfoot, something went wrong :-/ | 10:31 |
| tristan | We have #! /bin/bash in ldd | 10:31 |
| tristan | baserock has #! /usr/bin/bash | 10:31 |
| ironfoot | wow, bash.bst looks a bit weird | 10:34 |
| tristan | ironfoot, yeah :-/ | 10:35 |
| tristan | I've been struggling with that, it happens in another thing I'm working with too | 10:36 |
| tristan | ruamel issue it seems | 10:36 |
| tristan | have to figure out the case and work around that | 10:36 |
| tristan | ironfoot, if you can figure out in what case the input produces that output, I would be grateful :) | 10:37 |
| ironfoot | :) | 10:37 |
| tristan | then maybe we can just tweak the string in that case and side-step that | 10:37 |
| *** leeming has quit IRC | 10:41 | |
| tristan | ssam2, might know actually | 10:41 |
| *** ssam2 has quit IRC | 10:41 | |
| *** tiagogomes has quit IRC | 10:41 | |
| ironfoot | tristan: so, baserock implemented the usr merge, creating symlinks in fhs-dirs | 10:47 |
| ironfoot | Then, the bit in bash.morph to make it go to /bin was removed | 10:48 |
| ironfoot | That's why works for them, but not for us | 10:48 |
| tristan | does it not work for us ? | 10:48 |
| tristan | ironfoot, well, I'm getting #! /bin/bash in that script, but I dont know that it 'doesnt work' | 10:49 |
| ironfoot | Fair | 10:49 |
| tristan | me will know momentarily for usbutils anyway | 10:50 |
| ironfoot | There isn't any /bin/bash in this build environment I was debugging before | 10:50 |
| tristan | Nod, so what exactly is the change ? | 10:51 |
| tristan | ironfoot, Am I out of date with fhs-dirs in build-essential ? | 10:51 |
| tristan | ironfoot, I suspect so, as it failed usbutils again | 10:55 |
| ironfoot | 7ae830f1bc4c355eaf76f99655160420edc520e9 in definitions | 10:55 |
| *** ssam2 has joined #buildstream | 10:55 | |
| ironfoot | not many changes really | 10:55 |
| *** leeming has joined #buildstream | 10:55 | |
| *** tiagogomes has joined #buildstream | 10:56 | |
| tristan | ssam2, do you recognize this kind of garbage 'occasionally' produced by ruamel: https://paste.gnome.org/ptzrwks57 ? | 10:58 |
| ssam2 | you mean the triple quoting ? | 10:59 |
| ssam2 | triple escaping rather ? | 10:59 |
| ssam2 | I guess the problem is it decided to put the whole line in " quotes | 10:59 |
| tristan | ssam2, yeah the escaping stringification | 10:59 |
| tristan | of something that should just appear human readable | 10:59 |
| ssam2 | but it already contains \\ quotes | 10:59 |
| ssam2 | that looks actually valid, but yeah it's nasty | 10:59 |
| ssam2 | you need to convince it to use the correct string representation | 11:00 |
| ssam2 | I can't remember off the top of my head how to do that, but it's possible | 11:00 |
| tristan | I was hoping to avoid that | 11:00 |
| tristan | by determining the input characteristic which causes that, and tweaking input to sidestep that research (if thats more convenient) | 11:00 |
| ssam2 | not sure.. i mean you pretty much never want to use " quoting for shell commands | 11:01 |
| tristan | who knows, maybe just prepending a newline would get us out | 11:01 |
| * ssam2 trying to remember what I used ruamel.yaml for in the past and whether I'll still have the code | 11:01 | |
| tristan | yeah, and that stringification almost never happens | 11:01 |
| tristan | just once in a rand(), ruamel pukes that up | 11:01 |
| ssam2 | right | 11:02 |
| tristan | in this case, we will want to have a custom "dumper" anyway | 11:03 |
| tristan | so that might solve it | 11:03 |
| tristan | I.e. with custom dumper we can do cool things like: Put the 'kind:' field at the top of the file | 11:03 |
| tristan | (conversion now gives random dict order) | 11:03 |
| ssam2 | shouldn't do... | 11:04 |
| ssam2 | that's the main purpose of the RoundTrip loader | 11:04 |
| tristan | ssam2, only if you loaded what you're saving | 11:04 |
| ssam2 | ah I guess so | 11:04 |
| ssam2 | are you setting default_style ? maybe passing default_style='plain' to the dumper would help? | 11:09 |
| tristan | currently using yaml.round_trip_dump(data, f) | 11:12 |
| tristan | lets see | 11:12 |
| tristan | hah, no that's worse | 11:13 |
| tristan | https://paste.gnome.org/pqtezb6iw | 11:13 |
| ssam2 | that's done the opposite of what I expected :-) | 11:31 |
| ssam2 | maybe 'plain' doesn't mean what I thought it did | 11:31 |
| ssam2 | fwiw in the migrations we pass default_flow_style=False | 11:33 |
| ssam2 | that was to avoid all the dicts being messed up, but seems that isn't happening in your case anyway | 11:33 |
| *** paulsher1ood has joined #buildstream | 11:34 | |
| paulsher1ood | did i hear someone say bst can now build gnome from baserock definitions? | 11:38 |
| tristan | paulsher1ood, _almost_ :) | 11:38 |
| * paulsher1ood is _almost_ excited, then :) | 11:38 | |
| tristan | Having an issue with usbutils, it looks like the manually converted build-essential needs to be updated for recent changes in baserock | 11:39 |
| tristan | the good news is, not one manual change so far in the converted output (beyond build-essential) is required | 11:39 |
| tristan | and should not have to be any | 11:39 |
| paulsher1ood | that is good news... kind of validates the whole premise (yaml ftw) | 11:40 |
| tristan | conversions are successful for two reasons: A.) Structured metadata (YAML) as you mention... and B.) Matching build topologies | 11:41 |
| tristan | ironfoot, do you know if that commit is the only one I need ? | 12:01 |
| tristan | ironfoot, odd, it didnt do what the commit message said it would | 12:04 |
| * tristan rechecks | 12:04 | |
| ironfoot | tristan: hehe, I wouldn't be surprised :P | 12:05 |
| paulsher1ood | tristan: does bst have an equivalent of kbas? | 12:08 |
| tristan | paulsher1ood, I intend to replace that with a readme of how to setup an ostree repo, but uploading to it is -ENOTIMPLEMENTED yet | 12:10 |
| tristan | it should be as easy as setting up an apache and hosting an ostree repo | 12:10 |
| paulsher1ood | that sounds harder than kbas :) | 12:11 |
| ssam2 | apache can be any web server | 12:12 |
| tristan | Well, setting up apache should be pretty universal knowledge | 12:12 |
| tristan | but indeed "any webserver" will do | 12:12 |
| ssam2 | the idea of OSTree was that the server doesn't require any special daemon, which means you can host an OSTree repo in Amazon S3 or anywhere | 12:12 |
| tristan | the hope is to have less moving parts, not more | 12:12 |
| ssam2 | because it's all just files | 12:12 |
| tristan | right, calling ostree --init-repo on a server and hosting it should be enough | 12:13 |
| tristan | except for the push part, we probably want to have it setup so that it recognizes your ssh public key | 12:13 |
| ssam2 | providing a script that sets up Python SimpleHTTPServer or something correctly would be handy, maybe | 12:16 |
| ssam2 | ostree test suite may already have something usable | 12:16 |
| ironfoot | it's going to be quicker to implement than kbas, for sure | 12:17 |
| * tristan thinks it's more attractive to say "just host this ostree repo" than to say "Install this custom software somewhere" | 12:22 | |
| paulsher1ood | +1 | 12:22 |
| ssam2 | yes, the script would just be for "i want to try this out quickly" usecase | 12:24 |
| ssam2 | generally if a project's readme starts with "set up Apache", I postpone doing so :-) | 12:25 |
| paulsher1ood | yup... apache would stop me, too | 12:26 |
| tristan | nod, a tiny python script is nice, with the headline "We only require that you host this ostree repo", and never mention apache | 12:28 |
| tristan | Another nice thing, is ostree automatically provides some integrated gpg support | 12:29 |
| tristan | not sure _exactly_ how we can set it up with people who push things, but it means we can gpg verify with a public key on the user's side, that the artifact he is downloading did not get mitm attacked | 12:30 |
| * tristan shakes fist at ironfoot and jjardon ... usr merge ! | 12:30 | |
| tristan | heh | 12:31 |
| tristan | looks like all other commits are build-essential unrelated, only I have to debug why it's not working | 12:31 |
| tristan | staging something that depends on stage2-fhs-dirs has /bin /sbin /lib /lib64 directories (not symlinks) | 12:32 |
| jjardon | tristan: mmm? | 12:34 |
| tristan | jjardon, everything beyond build-essential was basically building with an auto-convert, except now I have to adjust my build-essential adaptation because, usr merge | 12:37 |
| ironfoot | tristan: not my fault :/ | 12:40 |
| tristan | nah, it's most likely mine for one reason or another | 12:41 |
| ironfoot | tristan: is there any kind of metadata in the filesystem? something like in baserock, the baserock/ folder? | 12:42 |
| tristan | nope | 12:43 |
| tristan | It may have to do with how we stage things... | 12:43 |
| tristan | and symlinks... | 12:43 |
| ironfoot | so, I've tried also to convert those changes to buildstream-tests | 12:43 |
| ironfoot | and yes, it only seems to be missing the symlinks | 12:43 |
| tristan | right I have a ported build-essential branch too | 12:44 |
| ironfoot | execvp /bin/sh: No such file or directory ? | 12:44 |
| ironfoot | well that's the error I have when fhs-dirs starts building | 12:45 |
| tristan | ironfoot, I have it on linux-api-headers, but I have some local changes staged in buildstream | 12:48 |
| tristan | namely, dont assume the location of the shell | 12:48 |
| tristan | stage2-fhs-dirs artifact is correct, staging it is incorrect | 12:58 |
| * tristan thinks he has fixed it | 13:31 | |
| tristan | but checking | 13:31 |
| * ironfoot wonders what was it | 13:32 | |
| tristan | ironfoot, it's all to do with staging and symlinks | 13:33 |
| tristan | which ironically, is something I spent last weekend fixing in ybd | 13:33 |
| ironfoot | but I thought that symlinks were needed in other parts to make it all work | 13:34 |
| tristan | (albiet, for a crazy system which puts symlinks in weird places, and stages broken symlinks to libraries inside libdirs) | 13:34 |
| tristan | ironfoot, not so far it seems | 13:34 |
| ironfoot | :) | 13:34 |
| ironfoot | ping me whenever you push the fix | 13:35 |
| tristan | Ok think I got it | 13:49 |
| * tristan stages commits | 13:49 | |
| tristan | mhmmm | 14:05 |
| tristan | Ok so, order of listed dependencies, should be meaningful ? or not meaningful ? | 14:05 |
| * tristan had followed juergbi's advice that they were not meaningful, allowing one to change the list without effecting cache keys | 14:06 | |
| tristan | hmm, yeah I think that is right | 14:08 |
| tristan | order of staging should be explicitly declared through dependencies | 14:08 |
| tristan | Which means, things which are meant to be staged *after* stage2-fhs-dirs, *must* depend on stage2-fhs-dirs | 14:08 |
| tristan | even if they dont really depend on that | 14:09 |
| tristan | I suppose, just "run" depend is enough for that expression, though | 14:09 |
| *** xjuan has joined #buildstream | 14:31 | |
| tristan | wow | 14:38 |
| tristan | Ok finally I got it | 14:38 |
| paulsher1ood | is 'wow' a good thing? | 14:39 |
| tristan | nope | 14:44 |
| tristan | wow is, I just spent all of sunday on symlinks, its a hard problem | 14:44 |
| tristan | and now I have to make the same fixes here | 14:45 |
| paulsher1ood | hah! | 14:45 |
| paulsher1ood | should be faster second time around :) | 14:45 |
| tristan | Yeah, and now discovered another problem :'( | 14:47 |
| tristan | this one was wrong for a while but it never mattered (or reared it's ugly face) until now | 14:48 |
| tristan | element.dependencies() returns direct dependencies first and then descends | 14:48 |
| tristan | which is wrong if what it depends on directly is also depended on by another dep later in the lst | 14:49 |
| tristan | list | 14:49 |
| * tristan thinks a proper tsort may be needed | 14:49 | |
| *** tristan has quit IRC | 14:58 | |
| *** tristan has joined #buildstream | 15:30 | |
| *** ChanServ sets mode: +o tristan | 15:30 | |
| *** csoriano has quit IRC | 15:56 | |
| xjuan | hmm for build deps, yeah you need a topological sort | 16:39 |
| xjuan | http://en.wikipedia.org/wiki/Topological_sorting | 16:39 |
| xjuan | https://git.gnome.org/browse/glade/tree/gladeui/glade-tsort.c | 16:39 |
| tristan | xjuan, :) | 16:39 |
| xjuan | :D | 16:39 |
| tristan | yeah it works properly already for build order, but not for stage order | 16:39 |
| tristan | that's an oversight | 16:39 |
| tristan | i.e. in buildstream jargon, staging means copying in the DESTDIR results of various make installs of each dependency | 16:40 |
| tristan | xjuan, and this only really makes a difference when 2 packages provide the same file, OR, when staging across symlink directory boundaries | 16:40 |
| tristan | so anyway, I've been wanting to unifiy some of the code, we have reliable build order, and we have a sort by depth | 16:42 |
| tristan | and we also have staging order | 16:42 |
| tristan | I want these to all be with the same loop | 16:42 |
| tristan | hopefully | 16:42 |
| tristan | will look into it tomorrow | 16:43 |
| xjuan | right | 16:44 |
| xjuan | so tsort usually has multiple valid results | 16:45 |
| xjuan | so if you want it to be consistent, you have to sort the input first | 16:45 |
| tristan | yes, but its desirable for the result to be constant | 16:45 |
| tristan | and the input is sorted at load time already :) | 16:45 |
| xjuan | ok thats good | 16:46 |
| tristan | but its also tricky because of build vs runtime dependencies | 16:46 |
| tristan | so, dependencies can be one, the other, or both | 16:46 |
| tristan | and only runtime dependencies are transitive | 16:46 |
| tristan | in any case thats not *too* tricky, but the dependencies for any given element need to be calculated separately | 16:47 |
| tristan | (cant have one tsort of everything) | 16:47 |
| tristan | (well that wouldnt make sense anyway) | 16:48 |
| tristan | in case it's of interest; this is what needs fixing: https://gitlab.com/tristanvb/buildstream/blob/master/buildstream/element.py#L109 | 16:50 |
| tristan | and this does something almost suitable: https://gitlab.com/tristanvb/buildstream/blob/master/buildstream/_pipeline.py#L44 | 16:50 |
| *** ssam2 has quit IRC | 18:25 | |
| *** xjuan has quit IRC | 23:07 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!