| *** valentind has quit IRC | 01:38 | |
| *** tpollard has quit IRC | 07:25 | |
| *** tiagogomes has quit IRC | 07:25 | |
| *** juergbi has quit IRC | 07:25 | |
| *** paulsherwood has quit IRC | 07:25 | |
| *** laurenceurhegyi has quit IRC | 07:25 | |
| *** nexus has quit IRC | 07:25 | |
| *** tpollard has joined #buildstream | 07:26 | |
| *** tiagogomes has joined #buildstream | 07:26 | |
| *** nexus has joined #buildstream | 07:26 | |
| *** juergbi has joined #buildstream | 07:26 | |
| *** paulsherwood has joined #buildstream | 07:26 | |
| *** laurenceurhegyi has joined #buildstream | 07:26 | |
| *** bochecha has joined #buildstream | 07:34 | |
| *** tristan has joined #buildstream | 07:51 | |
| *** jude has joined #buildstream | 08:12 | |
| juergbi | pytest-xdist looks interesting | 08:18 |
|---|---|---|
| juergbi | ./setup.py test --addopts "-n auto" | 08:19 |
| juergbi | 619 passed, 16 skipped in 20.92 seconds | 08:19 |
| juergbi | instead of over two minutes | 08:19 |
| tristan | juergbi, nice ! | 08:40 |
| tristan | juergbi, we need to require that extra thing in tests_require for it ? | 08:40 |
| tristan | or just right now, just add -n auto ? | 08:40 |
| juergbi | i added it to tests_require | 08:41 |
| juergbi | not sure whether it would work without that if pytest-xdist was already installed | 08:41 |
| tristan | nah doesnt | 08:41 |
| juergbi | installing pytest-xdist without specifying '-n auto' doesn't seem to have any side effects, i.e., still runs single threaded | 08:42 |
| tristan | without it it bails out saying missing extension or similar | 08:42 |
| juergbi | shall i push this dep? | 08:42 |
| tristan | doesnt work for me | 08:42 |
| juergbi | hm, after adding pytest-xdist to tests_require? what error do you get? | 08:43 |
| tristan | juergbi, I get a ton of these: https://bpaste.net/show/3bc962be957c | 08:43 |
| tristan | nonstop those | 08:43 |
| tristan | maybe it expects things to all be installed system wide when it finds some things installed systemwide, I have no idea | 08:44 |
| juergbi | hm, maybe some issue mixing different python package installation mechanisms? | 08:44 |
| tristan | I try to keep most of my things in ~/.local | 08:44 |
| juergbi | i see you have /usr/local and ~/.local for python | 08:44 |
| tristan | yes python itself is from /usr/local | 08:44 |
| tristan | and I try to make sure anything I install with pip is with --user | 08:44 |
| tristan | so it lands in ~/.local | 08:44 |
| juergbi | ./setup.py installed pytest-xdist in .eggs for me | 08:44 |
| tristan | and I can wipe it if desired | 08:44 |
| juergbi | and that just works | 08:44 |
| tristan | yes it did that for me too | 08:45 |
| tristan | in my .eggs, I have coverage, pep8, pytest_forked and pytest_xdist now | 08:45 |
| juergbi | i also have pytest_pep8 and a few more | 08:46 |
| juergbi | pytest itself is also in .eggs | 08:46 |
| tristan | oh I also have pytest_env and pytest_runner | 08:47 |
| tristan | but pytest itself I think is installed in ~/.local | 08:47 |
| tristan | in any case, looks a bit buggy | 08:47 |
| tristan | juergbi, I suspect if you uninstall buildstream, git clean -xdf, pip install --user pytest, ./setup.py test... you might reproduce | 08:48 |
| juergbi | i don't currently use pip. only system/distro + local .eggs | 08:48 |
| tristan | juergbi, if it's fragile, it deserves documentation if we're going to add it; which probably belongs in HACKING.bst | 08:49 |
| juergbi | i'm probably not the best person to debug this | 08:49 |
| tristan | HACKING.rst | 08:49 |
| tristan | the bst typos begin ! | 08:49 |
| juergbi | as long as it doesn't break the standard test invocation, i guess we could still add it to setup.py | 08:49 |
| * tristan recalls forever writing such things as "I'm so glade that happened !" | 08:49 | |
| juergbi | and then describe the parallel invocation with caveats in HACKING.rst | 08:49 |
| juergbi | haha | 08:49 |
| juergbi | muscle memory | 08:50 |
| tristan | juergbi, the standard test invokation is hard to describe really, nothing says that users have not previously installed pytest to run it manually against their other python projects | 08:50 |
| tristan | or whatever | 08:50 |
| tristan | this looks like a bug in pytest-xdist to be honest | 08:50 |
| juergbi | yes, the whole distro vs pip/egg/setuptools is just a mess, imo | 08:51 |
| tristan | juergbi, maybe we could describe a "standard dev setup" somehow, which might include using a virtual env ? | 08:51 |
| tristan | and then get something half way trustable | 08:51 |
| tristan | right now though, `./setup.py test` seems to be working with every combination of what is installed where | 08:52 |
| juergbi | maybe we should just use a buildstream-built environment for buildstream ;) | 08:52 |
| tristan | things can be in /usr/local, in ~/.local, and in .eggs/, and all is working | 08:52 |
| tristan | yeah, I know that is partly sarcasm | 08:53 |
| tristan | that would be painting us into a little corner indeed | 08:53 |
| tristan | "buildstream broke, I cant run tests because I cant produce the environment I need !" | 08:53 |
| * tristan 's laptop has become the default autobuilder for gnome :-/ | 08:55 | |
| paulsherwood | while i'm all in favour of determinism in the environment, this is one thing that hurt baserock badly originally... | 08:55 |
| tristan | I've been meaning to setup that intel box but cant find the time | 08:55 |
| paulsherwood | we required a baserock vm in which to run baserock... and as a result the setup time was woeful | 08:55 |
| tristan | paulsherwood, yes I was thinking of exactly that :) | 08:55 |
| paulsherwood | is vagrant a possibility? | 08:56 |
| * tristan doesnt know... | 08:56 | |
| tristan | I need to see just how immediately and without hassle I can just clone buildstream and run ./setup.py test | 08:57 |
| tristan | and I dont know anything about vagrant :) | 08:57 |
| paulsherwood | neither did i, but i found it relatively easy to setup for ybd | 08:57 |
| juergbi | we already have the docker image | 08:57 |
| tristan | paulsherwood, note this is specifically about running `make check` style tests, and speeding them up with this pytest-xdist extension | 08:57 |
| paulsherwood | tristan: ack, ok | 08:57 |
| tristan | paulsherwood, beside that, we already have a very deterministic env for running the tests on gitlab | 08:58 |
| paulsherwood | ok... would be nice to be able to keep doing everything outside gitlab too, to avoid spof etc | 08:58 |
| tristan | That is a point, we are already quite close to that though | 08:59 |
| tristan | integration-tests directory needs refactor but for other reasons, you can run the tests on your machine | 08:59 |
| tristan | and we do have the `bst-here` script which spawns the docker image, which I think is the same one we use for tests | 08:59 |
| tristan | so *probably* that same test run is reproducible on any machine | 08:59 |
| tristan | or quite close to it | 09:00 |
| tristan | on the other hand, I like that gitlab is free software and we can set it up at will without relying on the org | 09:00 |
| tristan | so I'm less worried about it slipping away | 09:00 |
| tristan | (well "free software", at least the parts we really need fall under that category) | 09:01 |
| paulsherwood | tristan: i agree, but other sponsored communities have disappeared before now | 09:01 |
| paulsherwood | i like gitlab a lot, to be clear. | 09:01 |
| tristan | similarly, we generate docs easily ourselves | 09:01 |
| tristan | but we rely on gitlab pipelines for things such as automatically doing the generation and updating a website | 09:02 |
| tristan | We do try to stay as self contained as possible; but I also think; if we're going to use things to automate stuff outside of that: I would much prefer to have "one thing" than many | 09:03 |
| paulsherwood | right. but nothing in those pipelines is magic :-) maybe we should sponsor a gitlab instance for this and other public-facing projects? :) | 09:03 |
| tristan | i.e. not gitlab plus jenkins plus vagrant plus.... | 09:03 |
| paulsherwood | totally agree | 09:03 |
| paulsherwood | in fact i keep trying to encourage additional functionality in gitlab, so we can avoid extra dependencies | 09:04 |
| paulsherwood | (we sponsored the original gitlab kanban work, for example) | 09:04 |
| tristan | yeah, I just do my part by filing bugs when I encounter them | 09:04 |
| paulsherwood | :) | 09:04 |
| tristan | I've been following this one particularly: https://gitlab.com/gitlab-org/gitlab-ce/issues/35141 | 09:05 |
| paulsherwood | bugs and discussions and documentation are contributions also... it's not all about code :-) | 09:05 |
| tristan | but looks like it wont be implemented very soon | 09:05 |
| tristan | yes, *definitely* agree, a huge amount of a project's value is in it's bug and email history | 09:06 |
| paulsherwood | +1 | 09:07 |
| * tristan notes that is why he likes to keep as much even marginally relevant bugs as possible open at all times | 09:08 | |
| tristan | even if it wont be fixed in 3 years, its valuable to remember that this little thing can be improved in some way | 09:08 |
| paulsherwood | yup | 09:09 |
| paulsherwood | i wonder if buildstream is 'ready' for me to try yet... :-) | 09:09 |
| tristan | paulsherwood, I think the bst-here script does what you wanted | 09:15 |
| tristan | http://buildstream.gitlab.io/buildstream/docker.html#docker | 09:16 |
| paulsherwood | ooh, cool! :-) | 09:16 |
| paulsherwood | i'll give it a try | 09:17 |
| *** jonathanmaw has joined #buildstream | 09:44 | |
| tristan | juergbi, oh... I thought about that and then forgot to mention... | 09:54 |
| tristan | juergbi, so... you have an itch and I rather not prevent you from scratching, while the change to setup.py is ridiculously trivial... | 09:55 |
| *** bethw has joined #buildstream | 09:55 | |
| tristan | juergbi, if you want to add it to tests_require, that doesnt really break things unless you add the --addopts thing | 09:55 |
| tristan | juergbi, so if you want, add it there, but please add a comment above the entry in tests_require, and mention that it's not entirely reliable | 09:56 |
| tristan | juergbi, but *on the other hand* | 09:56 |
| tristan | juergbi, first push a branch with that and make sure it doesnt break the docker builds, docker might need an update for this | 09:57 |
| tristan | because we try to take special care when running tests from docker that we get no surprise versions of things | 09:57 |
| juergbi | ok, i'll add it to my wip branch and will verify that it doesn't break the docker builds | 09:58 |
| *** tpollard has quit IRC | 10:07 | |
| *** ssam2 has joined #buildstream | 10:08 | |
| *** tpollard has joined #buildstream | 10:13 | |
| tristan | juergbi, if it breaks, you need to update: https://gitlab.com/BuildStream/buildstream-docker-images/blob/master/fedora/prepare.sh | 10:16 |
| juergbi | ok, ta | 10:16 |
| gitlab-br-bot | push on buildstream@juerg/recursive-pipelines (by Jürg Billeter): 9 commits (last: setup.py: Add pytest-xdist dependency) https://gitlab.com/BuildStream/buildstream/commit/72aeaba142157aa4788ccafd3ea8c24bd4a494c0 | 10:32 |
| *** valentind has joined #buildstream | 10:36 | |
| gitlab-br-bot | buildstream: merge request (sam/canonical-push-urls->master: Canonical push/pull URLs) #158 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/158 | 10:56 |
| tristan | ssam2, sure write a letter ! :D | 10:58 |
| ssam2 | ok! | 10:58 |
| tristan | ssam2, I didnt look at the patch at all yet, but it's good to get that expected breakage done in advance of 1.0 anyway :) | 10:58 |
| tristan | I'll take a look | 10:58 |
| tristan | are there a lot of edge cases ? I guess there is no such thing as having ssh: access for pull but not push | 10:59 |
| tristan | requires the same permissions | 10:59 |
| ssam2 | yes | 10:59 |
| tristan | I wish we had tests for the protocol :-S | 10:59 |
| ssam2 | if you try and use ssh:// to access a cache that's not been updated, pulls and pushes fail | 10:59 |
| ssam2 | but that's on us to update them :-) | 11:00 |
| tristan | Right, do they fail gracefully ? | 11:00 |
| ssam2 | yes | 11:00 |
| tristan | protocol version check I guess | 11:00 |
| ssam2 | "Remote cache did not tell us its pull URL. This cache probably requires updating to a newer version of `bst-artifact-receive`." | 11:01 |
| ssam2 | i didn't bump the protocol version, as I didn't add any new commands | 11:01 |
| ssam2 | we could do that, instead of having a check for whether it returned pull-url in the 'info' response. | 11:02 |
| gitlab-br-bot | buildstream: issue #160 ("pushing of artifacts should be imperative when enabled during bst build") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/160 | 11:05 |
| ssam2 | yes protocol tests would be kinda nice | 11:12 |
| tristan | thats where we're missing coverage a lot | 11:13 |
| ssam2 | there are a few options on how to do it ... one is to run our own SSH daemon somehow | 11:13 |
| tristan | still lagging barely below 80% | 11:13 |
| ssam2 | if you were happy for the test suite to depend on Paramiko, that would be the best way I think | 11:13 |
| tristan | I dont know what paramiko is | 11:13 |
| ssam2 | http://www.paramiko.org/ | 11:14 |
| ssam2 | we could have an in-process SSH daemon | 11:14 |
| ssam2 | running in the tests, calling the real bst-artifact-receive code | 11:14 |
| ssam2 | alternately, we could depend on the host's sshd, but that seems more fragile | 11:14 |
| ssam2 | or, we could just connect bst-artifact-receive to a pipe or a tcp socket... but then we need to add a new URL scheme just for the tests | 11:14 |
| ssam2 | we'd have an artifact cache url of unix:///tmp/bst-test-artifact-receive or something | 11:15 |
| ssam2 | (i guess that'd be a unix domain socket, not a tcp socket, but you get my point) | 11:15 |
| tristan | yeah... would it run through the same ssh remote commands codepath ? | 11:15 |
| ssam2 | yeah, it would | 11:16 |
| tristan | the point would to make the tests do the real thing of course | 11:16 |
| ssam2 | we'd use the host's `ssh` program, and that would talk to the test-suite's ssh daemon which would be provided by Paramiko | 11:16 |
| ssam2 | so we'd depend on the host having ssh but not sshd | 11:16 |
| tristan | yeah that sounds more real than in process ssh daemons | 11:16 |
| tristan | oh, well... why paramiko at all ? I dont understand why :-S | 11:17 |
| tristan | I mean, we always use sshd in real life right ? | 11:17 |
| ssam2 | sure | 11:17 |
| ssam2 | but we require a human to configure it | 11:17 |
| ssam2 | if we configure it automatically in the test suite, we basically have to require that it's opensshd | 11:18 |
| ssam2 | maybe that's fine | 11:18 |
| ssam2 | but it seems ugly | 11:18 |
| tristan | Ah, so... I mean, it's not a hard requirement to have it run in `./setup.py test` | 11:18 |
| *** cs_shadow has joined #buildstream | 11:18 | |
| tristan | it can be an integration test only | 11:18 |
| tristan | and will still be fine | 11:18 |
| ssam2 | good point... maybe that's fine then | 11:18 |
| ssam2 | you can run opensshd as a regular user with a custom config fine | 11:19 |
| tristan | ssam2, so we can't like... provide a stock sshd_config that runs in an isolated container or smth ? | 11:19 |
| tristan | I guess that's getting a bit too much | 11:19 |
| tristan | or that | 11:19 |
| tristan | in any case, the integration tests *should* still be runnable on the host so, I'm not sure what I was thinking | 11:20 |
| tristan | well, it's certainly a tricky thing to do, if paramiko makes it easier then that sounds like a good approach | 11:21 |
| tristan | ssam2, also note that we can optionally enable / disable tests quite easily (and do so quite a bit) | 11:21 |
| ssam2 | yeah true | 11:21 |
| tristan | So, we dont need to hard depend on it | 11:21 |
| ssam2 | and i think nearly everyone does use opensshd in practice | 11:21 |
| tristan | just make sure the docker image has it so that it runs on the main gitlab pipelines | 11:22 |
| ssam2 | perhaps somewhere like Solaris you have something incompatible | 11:22 |
| tristan | another tricky thing to this... | 11:24 |
| tristan | is that when running bst-artifact-receive in a test harness, it needs to be run under coverage as well | 11:24 |
| ssam2 | hmm yeah | 11:24 |
| tristan | and the report collected safely separately | 11:24 |
| ssam2 | it's up to the test runner to configure the ssh daemon to do that, I guess | 11:25 |
| tristan | ssam2, anyway; I'm not blocking your patch on that - but having tests for that part is still a big need | 11:25 |
| tristan | so *cough*, if anyone is going to work on that, I'm very pleased :D | 11:26 |
| ssam2 | maybe :-) | 11:27 |
| ssam2 | but also want to look into the slowness i found yesterday :-( | 11:27 |
| tristan | if that fsync variable works, we can add it in GNOME | 11:28 |
| tristan | maybe specifically for that integration command | 11:28 |
| ssam2 | yes, it does seem to help in some cases | 11:28 |
| tristan | doesnt seem like a good candidate for the default environment | 11:28 |
| ssam2 | agreed | 11:28 |
| ssam2 | there's also an `update-mime-database -n` flag which checks timestamps and does no work if it's not needed | 11:29 |
| ssam2 | i'm wondering if we are messing up timestamps in the integration stage and losing some optimisations as a result | 11:29 |
| ssam2 | ldconfig is also super slow, for every single element, which seems wrong | 11:29 |
| ssam2 | if we run that after building the base, it should have nothing to do when run again | 11:29 |
| ssam2 | but maybe timestamps are causing it to redo everything, each time it runs | 11:30 |
| ssam2 | there's clearly another issue as well, because none of that would cause checkouts to be super slow. I tried disabling ostree's fsync's and that didn't help, so it may well be something FUSE related | 11:30 |
| ssam2 | what i actually wanted to do was write a talk and do some screencasts of using BuildStream | 11:31 |
| tristan | ssam2, can try adding ldconfig in base-configure.bst | 11:44 |
| tristan | but we will lose the timestamps anyway | 11:44 |
| tristan | if they count | 11:44 |
| tristan | at *least* until we figure out something for #38 | 11:44 |
| * tristan starting to remember that infamous bug number by heart now | 11:45 | |
| ssam2 | right | 12:00 |
| ssam2 | adding ldconfig in base-configure might help a lot | 12:00 |
| tristan | I dont know that it will, but it might | 12:13 |
| tristan | :) | 12:13 |
| tristan | I honestly suspect it will not make a difference | 12:14 |
| ssam2 | where would i file a bug for this ? | 12:14 |
| ssam2 | buildstream project, jhbuild-bst project ? | 12:14 |
| tristan | I dont see how it's possible that `dpkg --configure -a` doesnt do it | 12:14 |
| ssam2 | hmm true | 12:14 |
| ssam2 | it did help during Baserock bootstrap work, but nothing already ran ldconfig there | 12:14 |
| tristan | Well, try it | 12:14 |
| tristan | dpkg --configure -a doesnt pass everything, some things fail | 12:15 |
| ssam2 | will do. i think i need to set some time aside for this though | 12:15 |
| tristan | maybe an extra kick in the nuts to the runtime loader will help | 12:15 |
| *** juergbi has left #buildstream | 12:15 | |
| *** juergbi has joined #buildstream | 12:15 | |
| juergbi | an alternative may be not to use ld.so.cache at all, might be faster overall | 12:16 |
| juergbi | (iirc, everything works if the cache doesn't exist, just without the cache speedup) | 12:16 |
| tristan | I feel like the slowness is due to the python fuse layer | 12:17 |
| tristan | to be honest | 12:17 |
| juergbi | that's possible | 12:17 |
| tristan | ssam2, you could try a build with that disabled; which is "dangerous" but might reveal the problem | 12:18 |
| juergbi | wondering whether overlayfs would be a potential (faster) alternative | 12:18 |
| tristan | I dont know, but I doubt that I want any external tools taking on that role | 12:18 |
| juergbi | overlayfs is in kernel mainline | 12:18 |
| tristan | because #38 *has* to happen | 12:18 |
| * tristan doesnt know the details of how overlayfs works | 12:19 | |
| tristan | maybe it also gives us the same kind of entry points that fuse does ? | 12:19 |
| tristan | that would give us the flexibility we need | 12:19 |
| tristan | but then, it would also lock us in more tightly with linux: bad | 12:19 |
| juergbi | no, overlayfs is an in-kernel union filesystem | 12:19 |
| juergbi | yes, it would have to be a linux-specific optimization, unfortunately | 12:20 |
| tristan | I have a feeling that a C implementation of the safe hardlinks thing might be faster | 12:20 |
| tristan | yeah the more we do linux specific stuff the more we grow platform entry points, the more mess we create | 12:20 |
| juergbi | overlayfs doesn't help with uid/gid handling but the also linux-specific user namespace support might | 12:20 |
| tristan | better to be a bit slower than to create something less standard and immensely difficult to maintain | 12:20 |
| juergbi | yes, i generally agree | 12:21 |
| ssam2 | i suspect FUSE, indeed | 12:27 |
| ssam2 | running a checkout with the python profiler would be my next step | 12:27 |
| tristan | ssam2, there has always been some consideration of categorizing integration commands a bit too, but that wouldnt make ldconfig faster, that's probably wanted all the time | 12:30 |
| tristan | the idea being, maybe only some integration commands are needed for builds, while others can be reserved for elements (and/or element types) which explicitly want them | 12:30 |
| tristan | but, you know all that jazz about premature optimizations... | 12:31 |
| tristan | so havent looked at it yet | 12:31 |
| ssam2 | yeah, it sounds a bit fiddly | 12:31 |
| ssam2 | these slowness issues are not unique to us; if a package manager had to wait 3 minutes running ldconfig after every operation, people would complain | 12:32 |
| ssam2 | when chasing the update-mime-database slowness i found various discussions on package manager bug trackers complaining of the same thing | 12:32 |
| tristan | Well, to be fair (or unfair, depending on how fast is your storage unit it seems), it's really much much faster than what you describe | 12:37 |
| tristan | heh | 12:37 |
| tristan | also I dont think dropping ldconfig is an option, or at least not an easy one; we do rely a bit on a custom ld.so.conf.d/ file to prioritize the libraries we install in the sysroot | 12:39 |
| tristan | looking at canonical urls: return (not self._offline or self._local) | 12:45 |
| tristan | that's not a change from before (i.e. before was just as ambiguous) | 12:46 |
| tristan | but I always wonder what does that boolean statement mean ? | 12:46 |
| tristan | (not a) or b | 12:46 |
| tristan | or | 12:46 |
| tristan | not (a or b) | 12:46 |
| ssam2 | yes, i copied that while trying not to think about it :-) | 12:49 |
| ssam2 | well, the only way it can work if it means: if (self._local) or (not self.offline): | 12:49 |
| gitlab-br-bot | push on buildstream@master (by Jürg Billeter): 1 commit (last: setup.py: Add pytest-xdist dependency) https://gitlab.com/BuildStream/buildstream/commit/841f4e9e47ed4d7d14d8526a657600a6a67d7312 | 12:49 |
| ssam2 | as being offline is a disaster, unless we are using a local path | 12:49 |
| ssam2 | so it means: (not a) or b | 12:49 |
| tristan | yeah, lets avoid such ambiguousness in general and stick with extra parens :) | 12:53 |
| tristan | but anyway, patch looks good, added comment | 12:53 |
| ssam2 | great! should i proceed with sending the mail, updating caches, and merging then ? | 12:53 |
| tristan | yeah, was going to ask you to just send an announcement about it | 12:53 |
| tristan | if you want to merge it today and update gnome7 and the baserock server, you can go ahead | 12:54 |
| tristan | for me, it's too late to stick around for that | 12:54 |
| tristan | otherwise, tomorrow we coordinate early morning UK time :) | 12:54 |
| ssam2 | i can merge | 12:55 |
| ssam2 | and do the stuff | 12:55 |
| ssam2 | progress! | 12:55 |
| tristan | :) | 12:57 |
| tristan | Yeah I'm happy to get that out of the way | 12:57 |
| tristan | all around nicer | 12:57 |
| * tlater can't figure out how to get buildstream to know the pipeline is consistent if only part of it is tracked :| | 13:11 | |
| tristan | tlater, in order to know if the lower, excepted levels are consistent/cached ? | 13:14 |
| tlater | tristan: Or the higher ones that aren't included in the tracking queue | 13:14 |
| tristan | tlater, that would need to be some more selective logic in how we initially load | 13:14 |
| tristan | i.e. there is a parameter to the Pipeline() constructor which is currently a boolean | 13:14 |
| tristan | whether to load inconsistent or not | 13:15 |
| tristan | that would have to be changed | 13:15 |
| tlater | Hm, not sure that would work too well, because we only know what is being tracked and what isn't after the Pipeline has loaded | 13:16 |
| tristan | tlater, so a goal here is to still be able to build without saving tracking results, and track, but except the base system, tracking the base system means downloading a few gigs you dont need | 13:16 |
| tristan | so, it would be important to know that | 13:16 |
| tristan | tlater, looking at _pipeline.py, we check inconsistent to selectively force elements to be inconsistent | 13:17 |
| tristan | and otherwise, we force calculate the cache key early | 13:17 |
| tristan | tlater, that whole thing could be migrated outside of the constructor ? | 13:18 |
| tlater | Right, actually, that should work | 13:18 |
| tristan | of course, it means aggressive changes | 13:18 |
| tristan | to the tickers and such | 13:18 |
| tristan | tlater, if I recall your main.py patch properly, that might even simplify main.py | 13:19 |
| tlater | A bit, probably. Well, I'll change that and see what breaks. | 13:20 |
| tristan | tlater, it might be easier to move that whole block outside of the constructor | 13:22 |
| * tlater was moving it into a separate function | 13:22 | |
| tristan | i.e. from preflighting all the way to checking cache state | 13:22 |
| tristan | not only checking cached state | 13:22 |
| tristan | all of that seems to belong together | 13:22 |
| tristan | and can be practically called from every main entry point | 13:23 |
| tristan | I dont know, you are currently closer to the code, it just *seems* that way | 13:23 |
| * tlater wonders about the workspace stuff, but agrees otherwise :) | 13:24 | |
| * tristan thinks it was a bad idea in general to have constructors "doing things" | 13:25 | |
| tristan | although I originally took that approach a lot, it has bitten me | 13:25 |
| *** mcatanzaro has joined #buildstream | 13:34 | |
| *** tristan has quit IRC | 13:47 | |
| *** semanticdesign has joined #buildstream | 13:49 | |
| ssam2 | i'm gonna merge the canonical push/pull URLs business then | 15:02 |
| gitlab-br-bot | push on buildstream@sam/canonical-push-urls (by Sam Thursfield): 20 commits (last: Fix tests) https://gitlab.com/BuildStream/buildstream/commit/486372e9d79641412214f2079112b5200bbc82d5 | 15:03 |
| gitlab-br-bot | buildstream: merge request (sam/canonical-push-urls->master: Canonical push/pull URLs) #158 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/158 | 15:04 |
| ssam2 | in a while, once the CI has finished | 15:04 |
| tlater | Hm, is there a way to remove an element from the cache in the tests? | 15:15 |
| ssam2 | several ways | 15:16 |
| ssam2 | do we always use the same backend in the tests, though ? | 15:17 |
| ssam2 | i assume we run each test twice in CI, once with each backend | 15:17 |
| ssam2 | so i guess this would need a helper function | 15:17 |
| tlater | For the test I'm trying to do this for I only use one backend | 15:19 |
| ssam2 | oh cool, so just delete it from the cache | 15:21 |
| ssam2 | if it's ostree, delete the appropriate ref | 15:21 |
| tlater | git? | 15:22 |
| tlater | Wait, no | 15:22 |
| tlater | Ah, | 15:22 |
| tlater | Alright | 15:22 |
| *** bochecha has quit IRC | 15:59 | |
| *** bochecha has joined #buildstream | 15:59 | |
| *** bochecha has quit IRC | 16:04 | |
| *** bochecha has joined #buildstream | 16:04 | |
| *** jude has quit IRC | 16:49 | |
| gitlab-br-bot | buildstream: merge request (132-loading-external-plugins-works-without-explicit-requirement-in-project-conf->master: WIP: Resolve "Loading external plugins works without explicit requirement in project.conf") #125 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/125 | 17:18 |
| gitlab-br-bot | push on buildstream@132-loading-external-plugins-works-without-explicit-requirement-in-project-conf (by Jonathan Maw): 1 commit (last: Make external plugin loading require explicit configuration in project.conf) https://gitlab.com/BuildStream/buildstream/commit/28711d2c00c7537736ae5d02c0f9e8cec9163bd6 | 17:19 |
| gitlab-br-bot | buildstream: merge request (132-loading-external-plugins-works-without-explicit-requirement-in-project-conf->master: WIP: Resolve "Loading external plugins works without explicit requirement in project.conf") #125 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/125 | 17:19 |
| *** bethw has quit IRC | 17:19 | |
| jonathanmaw | ouch! after making those changes 250 of the pytest tests failed | 17:20 |
| tlater | Oof | 17:21 |
| ssam2 | the unix platform tests are unbelievably slow these days | 17:36 |
| ssam2 | i don't think i'm even going to get time to merge the canonical pull URLs branch today at this rate | 17:37 |
| ssam2 | i guess i can spend the time trying to work out why it's so slow. seems to be a theme of this week | 17:37 |
| tlater | ta ssam2 for solving the slowness mysteries :) | 17:39 |
| gitlab-br-bot | push on buildstream@tracking-changes (by Tristan Maat): 2 commits (last: _pipeline.py: Merge the track queue into the scheduler run) https://gitlab.com/BuildStream/buildstream/commit/09c334030ef527a1c1a7bf957a15f04fc00bbd26 | 17:40 |
| gitlab-br-bot | buildstream: merge request (tracking-changes->master: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 17:40 |
| jonathanmaw | only 25 failed now, tests pointed out where I was loading the project wrong | 17:48 |
| gitlab-br-bot | push on buildstream@tracking-changes (by Tristan Maat): 8 commits (last: setup.py: Add pytest-xdist dependency) https://gitlab.com/BuildStream/buildstream/commit/841f4e9e47ed4d7d14d8526a657600a6a67d7312 | 17:50 |
| gitlab-br-bot | buildstream: merge request (tracking-changes->master: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 17:50 |
| gitlab-br-bot | push on buildstream@tracking-changes (by Tristan Maat): 1 commit (last: Fix tests) https://gitlab.com/BuildStream/buildstream/commit/d1dc2286cc42dcaab35f219f54f4b612e6ae9c02 | 17:53 |
| gitlab-br-bot | buildstream: merge request (tracking-changes->master: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 17:53 |
| *** jonathanmaw has quit IRC | 18:04 | |
| *** ssam2 has quit IRC | 18:06 | |
| *** bethw has joined #buildstream | 18:22 | |
| *** tristan has joined #buildstream | 19:06 | |
| *** bethw has quit IRC | 19:53 | |
| *** tristan has quit IRC | 21:41 | |
| *** bethw has joined #buildstream | 21:58 | |
| *** valentind has quit IRC | 22:39 | |
| *** bethw has quit IRC | 23:07 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!