| *** tristan has joined #buildstream | 02:58 | |
| gitlab-br-bot | buildstream: issue #271 ("Assert minimum version of bwrap on host") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/271 | 05:22 |
|---|---|---|
| gitlab-br-bot | buildstream: merge request (tristan/versioneer->master: Use versioneer instead of setuptools_scm) #436 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/436 | 05:28 |
| gitlab-br-bot | buildstream: merge request (tristan/versioneer->master: Use versioneer instead of setuptools_scm) #436 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/436 | 05:43 |
| tristan | https://gitlab.com/BuildStream/buildstream/merge_requests/436 Code quality degrated on 77 points ! | 05:48 |
| tristan | jjardon, ^^^ :) | 05:48 |
| tristan | that is kinda cute | 05:48 |
| gitlab-br-bot | buildstream: merge request (tristan/uniform-junction-errors->master: Uniform junction errors) #437 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/437 | 05:50 |
| gitlab-br-bot | buildstream: merge request (tristan/uniform-junction-errors->master: Uniform junction errors) #437 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/437 | 06:05 |
| gitlab-br-bot | buildstream: merge request (tristan/dont-install-tests->master: setup.py: Stop installing test cases.) #438 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/438 | 06:13 |
| gitlab-br-bot | buildstream: merge request (tristan/dont-install-tests->master: setup.py: Stop installing test cases.) #438 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/438 | 06:37 |
| *** ChanServ sets mode: +o tristan | 06:52 | |
| *** tristan changes topic to "/BuildStream 1.1.3 is out ! | https://gitlab.com/BuildStream/buildstream | Docs: https://buildstream.gitlab.io/buildstream | IRC logs: https://irclogs.baserock.org/buildstream | Roadmap: https://wiki.gnome.org/Projects/BuildStream/Roadmap" | 06:52 | |
| *** tristan has quit IRC | 06:56 | |
| *** bochecha_ has joined #buildstream | 07:39 | |
| *** bochecha has left #buildstream | 07:39 | |
| valentind | The case of bzr for mirror namespaces is a bit special. The reference is just number not a hash. So if we have a case where history has been rewritten and we have several mirrored repositories, when we look for a revision, there is no way to well identify which one it is, and it will always take it from the oldest repository. | 07:44 |
| *** bochecha_ has quit IRC | 07:53 | |
| *** bochecha_ has joined #buildstream | 07:53 | |
| *** bethw has joined #buildstream | 08:06 | |
| *** toscalix has joined #buildstream | 08:08 | |
| *** dominic has joined #buildstream | 08:31 | |
| jjardon | Tristan :) | 08:43 |
| jjardon | Congrats for the release all! | 08:43 |
| * jjardon wonders why the version at buildstream.gitlab.io/ has not changed | 08:44 | |
| *** jonathanmaw has joined #buildstream | 08:51 | |
| paulsherwood | is bzr still a thing? | 08:51 |
| jmac | Yes | 08:53 |
| *** bochecha_ has quit IRC | 08:58 | |
| *** bochecha_ has joined #buildstream | 09:03 | |
| jmac | Having said that, I can't find any projects still using it for primary source control | 09:03 |
| paulsherwood | i had thought canonical de-funded but maybe i misunderstood | 09:04 |
| dominic | heh, it's apparently a mix of charity and mugger | 09:04 |
| paulsherwood | ? | 09:04 |
| dominic | oops, w/w | 09:05 |
| bochecha_ | paulsherwood: canonical stopped funding it, yes | 09:05 |
| bochecha_ | but most project on Launchpad still use bzr | 09:05 |
| bochecha_ | not all have moved to Git | 09:06 |
| paulsherwood | bochecha_: ack, thanks | 09:06 |
| * paulsherwood has been telling people that the VCS war was won nearly a decade ago, but that's clearly an oversimplification :) | 09:06 | |
| bochecha_ | it's not… Git won a while ago | 09:07 |
| bochecha_ | some people just haven't received the memo yet ;) | 09:07 |
| paulsherwood | lol | 09:07 |
| bochecha_ | bzr is also python2-only iirc, so it will get removed from distros eventually :P | 09:07 |
| bochecha_ | that's going to be fun times, when some upstream maintainers of active projects can't pull/push their code any more >_< | 09:07 |
| * paulsherwood wonders whether that will be before or after the 2038 fun times | 09:08 | |
| jmac | There are plenty of important projects still officially using SVN, and at least one on BitKeeper | 09:10 |
| bochecha_ | jmac: SVN is still maintained upstream though | 09:10 |
| *** tristan has joined #buildstream | 09:18 | |
| valentind | Strangely when I run tests on a branch I get failures in get_bst_version(), because it tries to read "0+untagged" as an integer. | 09:26 |
| valentind | tristan, you were not here so I will repeat what I said before: The case of bzr for mirror namespaces is a bit special. The reference is just number not a hash. So if we have a case where history has been rewritten and we have several mirrored repositories, when we look for a revision, there is no way to well identify which one it is, and it will always take it from the oldest repository. | 09:28 |
| tristan | valentind, I see | 09:30 |
| tristan | valentind, for bzr I see... it's indeed hard to authenticate, not sure what we can practically do about it | 09:30 |
| tristan | in the absence of a solution, no solution is acceptable :) | 09:31 |
| tristan | even if there is a complex possible solution, no solution is also acceptable; these are things we can hopefully improve over time | 09:31 |
| tristan | valentind, what is the failure you get for get_bst_version() ? | 09:31 |
| tristan | I had fixed a test before pushing the migration of setuptools_scm -> versioneer this morning | 09:32 |
| tristan | it's certainly related | 09:32 |
| tristan | but afaics my tests have been passing | 09:32 |
| valentind | File "/home/valentin/repos/buildstream/buildstream/utils.py", line 475, in get_bst_version | 09:33 |
| valentind | return (int(versions[0]), int(versions[1])) | 09:33 |
| valentind | ValueError: invalid literal for int() with base 10: '0+untagged' | 09:33 |
| tristan | How do I reproduce this ? | 09:34 |
| valentind | When running bst installed, it works fine. But when running from the tests it fails. | 09:34 |
| valentind | I just have a branch, and then I run the tests, e.g. 'python3 setup.py test --addopts "tests/sources/zip.py::test_no_ref"' | 09:35 |
| tristan | Ummm | 09:35 |
| tristan | Hmmm | 09:35 |
| tristan | valentind, so I have to create a branch, make a modification, and then run tests ? | 09:35 |
| valentind | tristan, This I am not sure. | 09:36 |
| * tristan is just unsure how you get "+untagged" | 09:36 | |
| tristan | I have been branching a few times and running tests, that is for sure | 09:36 |
| valentind | No, actually it fails for me on master | 09:37 |
| valentind | When I run "python3.6 setup.py build" | 09:37 |
| valentind | I get at the end: | 09:37 |
| valentind | set build/lib/buildstream/_version.py to '0+untagged.2296.g06ae434' | 09:37 |
| tristan | 3.6 ! | 09:39 |
| tristan | hmm, I hope this is not python version related, that would be weird | 09:40 |
| tristan | If I run `python3 setup.py build`, I get: set build/lib/buildstream/_version.py to '1.1.3' | 09:41 |
| valentind | I think it is because I have no tag in my clone. | 09:41 |
| tristan | at the end :-/ | 09:41 |
| tristan | I did push the tag though right ? | 09:41 |
| tristan | I recall pushing the tag | 09:41 |
| valentind | Yes, I did a git fetch. | 09:41 |
| tristan | and anyway, you should be getting 1.1.2 if not | 09:41 |
| valentind | Problem is when you just do "git pull origin master". | 09:41 |
| valentind | Then you do not have the tags. | 09:41 |
| tristan | Errrm | 09:42 |
| * skullman tends to `git remote update` | 09:42 | |
| tristan | Ok, is it typical that this actually happens ? | 09:42 |
| * tristan does `git pull --rebase` usually | 09:42 | |
| tristan | valentind, I think in this case, we probably need an error | 09:42 |
| tristan | valentind, would be good if you could file an issue about this please | 09:43 |
| valentind | OK | 09:43 |
| tristan | I would say that, either you should have the git clone with bells and whistles... or you should have a dist tarball, but there is no point to support something in between | 09:44 |
| *** aday has joined #buildstream | 09:44 | |
| valentind | for tag in $(git tag); do git tag --delete ${tag}; done | 09:44 |
| tristan | (so probably we need a better runtime error) | 09:44 |
| valentind | To reprodocue | 09:44 |
| tristan | haha | 09:44 |
| tristan | no thanks, I really believe you | 09:44 |
| tristan | valentind, a better reproduce is: "Clone buildstream like this: git ..." | 09:45 |
| gitlab-br-bot | buildstream: issue #380 ("Uncaught ValueError when tags are not pulled into local repository") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/380 | 09:52 |
| juergbi | jonathanmaw: I'm a bit confused with !317. For tracking we have redirection in _get_elements_to_track() and in track(). Do we really need it in the latter as well? Isn't this already covered by the former? | 09:55 |
| *** bethw has quit IRC | 09:57 | |
| juergbi | In the former it's only for deps none and without redirection message but the combination still confuses me | 09:57 |
| jonathanmaw | juergbi: I'm not sure. It's possible that it's because of `bst workspace open --track`, but I'll have to check the details | 09:59 |
| * jonathanmaw has a meeting now, will look at it when I get back | 09:59 | |
| juergbi | ok | 10:00 |
| jennis | Hi guys, I've written a test which builds two 5 MB artifacts and pushes to remote cache with a 12 MB quota. Element 1 is then removed from the local cache. Element1 is then pulled from the remote cache. A third (5 MB) element is then built and pushed to the remote cache. Ideally, considering LRU, we would like Element1 and Element3 to remain in the cache. | 10:20 |
| jennis | However, `bst pull` does not update the mtime of the artifact, and this is what we use to sort LRU artifacts | 10:21 |
| jennis | Also, invoking `bst pull` eventually ends up with OSTree.Repo.pull being executed from the client side | 10:21 |
| jennis | So I'm struggling to find a way to update the mtime of an artifact on the server just before it is pulled | 10:22 |
| jennis | Any ideas would be appreciated | 10:22 |
| juergbi | jennis: iirc, we discussed that we won't be able to implement LRU for this initial solution. it will be least recently pushed/modified instead | 10:23 |
| juergbi | it's still much better than just wiping the whole artifact cache | 10:23 |
| juergbi | with CAS we will have more control and should be able to switch to LRU relatively easily | 10:23 |
| jennis | Ok, that's what is currently implemented then | 10:24 |
| jennis | It is much better | 10:24 |
| jennis | I just worry this will end up with more rebuilds of expensive artifacts | 10:25 |
| jennis | e.g. a base system which may be pulled frequently (this would be top of the least recently pushed list) | 10:25 |
| jennis | would likely be* | 10:25 |
| juergbi | yes, this is indeed a concern, but LRU is not easy to accomplish with OSTree and we need a solution for running out of disk space now | 10:26 |
| juergbi | LRU will be a follow up task, but we might defer it until after the switch to CAS | 10:26 |
| jennis | Ok, I will not deal with that now then | 10:27 |
| *** bethw has joined #buildstream | 10:27 | |
| jennis | juergbi, would you like me to leave the test (commented out) in the MR? | 10:27 |
| jennis | So that when we do come to dealing with it, we don't need to write it again... | 10:27 |
| juergbi | We should definitely not throw it away but not sure how to handle it best. We could also put it in a separate branch but we must not forget about it. We could link to the branch from the gitlab issue | 10:29 |
| jennis | ok, I'll make a separate branch and link it. I'll also save it locally and hopefully remember when I see this come up in the future :p | 10:32 |
| tlater | juergbi, jennis: Use @pytest.skip or @pytest.xfail | 10:52 |
| tlater | That's literally the purpose of those :) | 10:52 |
| juergbi | xfail would make sense. assuming we handle those properly in CI etc. | 10:54 |
| tlater | They are handled by pytest, so we *should*. Interesting experiment in either case. | 10:56 |
| tlater | jennis: Btw, to alleviate your concerns of removing expensive artifacts; once a client has downloaded an artifact it will keep reusing it, because it will be cached locally. | 11:14 |
| tlater | So in the absolute worst case, you will end up just building that artifact once for each user. | 11:14 |
| tlater | Which isn't horrifyingly bad, IMO. | 11:14 |
| jonathanmaw | juergbi: I'm still working on figuring out exactly what's happening, but it looks like the redirection in Pipeline.track() isn't redundant. When I remove it, I'm seeing tests failing because elements that weren't scheduled for tracking are ending up in a track queue | 11:21 |
| tristan | Do I understand that remote expiry is coming soon to a cache server near me ? | 12:50 |
| tristan | Starring James Ennis and the Artifacts ? | 12:50 |
| tristan | This is *awesome* | 12:51 |
| tristan | Untracked bugfix of the day: setuptools_scm was only ever updating the installed version at setup.py install time, which happens almost never for the majority of our users; who report issues and "helpfully" indicate the version they are using | 12:53 |
| tristan | replaced with similar thing called "versioneer", and that is fixed, snuck into 1.1.3 | 12:54 |
| tristan | upgrades also smoothly start using the thing, so upgrading to 1.1.3 with `git pull` requires no re-install | 12:54 |
| *** xjuan has joined #buildstream | 12:57 | |
| tristan | milloni, By the way... I noticed that you dont have a name when doing the release notes this morning (which would be in the middle of the night in european places), wanted to ask you but you were away | 12:58 |
| tristan | milloni, I would recommend using `git config` such that your commits have your name on them :) | 12:59 |
| tristan | of course, I have no qualms if you prefer to remain an anonymous (or pseudonymous) contributor :) | 12:59 |
| tristan | milloni the mysterious :) | 13:00 |
| jennis | tristan, yes :) | 13:02 |
| tristan | jennis, awesome, we've been seeing 20hour build pipelines in gnome-build-meta | 13:03 |
| tristan | with remote artifact expiry we can start to actually *enable* remote caching | 13:04 |
| tristan | coupled with junction.refs separation, means we will also build less often the freedesktop-sdk project | 13:04 |
| tristan | which should bring builds down to the expected norm | 13:04 |
| skullman | GitLab seems to be unusually slow today | 13:15 |
| tristan | Sounds familiar :) | 13:15 |
| tlater | skullman: They are actually having infrastructure issues | 13:15 |
| tlater | Or, at least, someone mentioned they did earlier this morning | 13:16 |
| skullman | good to know it's not just me then | 13:16 |
| tristan | it seems to vary from project to project, too | 13:19 |
| tristan | freedesktop-sdk was noticeably more laggy than buildstream last night | 13:21 |
| tristan | last couple of weeks have been difficult, but it's certainly an improvement over last year | 13:22 |
| tristan | where it would be unreachable for almost a day, rather frequently | 13:23 |
| *** tristan has quit IRC | 13:40 | |
| *** xjuan has quit IRC | 13:41 | |
| *** tristan has joined #buildstream | 14:01 | |
| jonathanmaw | juergbi: after looking into it in a bit more detail, the duplication in Pipeline.track() and Pipeline._get_elements_to_track() is currently required because _get_elements_to_track only redirects if mode=PipelineSelection.NONE. The simple (but confusing) way to fix this would be to pass `track_selection="none"` into the args in cli.py:workspace_open. The proper way is probably to add the arg "redirect_track_elements" to Pipeline.initialize() (an | 14:02 |
| jonathanmaw | d all the intervening function signatures up to the cli) | 14:02 |
| *** cs_shadow has joined #buildstream | 14:06 | |
| juergbi | jonathanmaw: why do you think setting track_selection=PipelineSelection.NONE for workspace_open would be wrong? | 14:09 |
| juergbi | we don't want any tracking of dependencies there, do we? | 14:10 |
| jonathanmaw | juergbi: true, but that's separate from its intended purpose | 14:10 |
| juergbi | to me this seems closely related | 14:11 |
| juergbi | redirection is pointless when anyway tracking dependencies as potential source elements will be included anyway | 14:12 |
| juergbi | it's only needed when we don't include dependencies | 14:12 |
| juergbi | and we don't want to include dependencies for tracking in workspace open, and redirection handling should work properly if we specify it that way | 14:13 |
| juergbi | am I misunderstanding something? | 14:13 |
| jonathanmaw | mostly, I'm worried that someone'll look at workspace_open() for example, and wonder why I bother to explicitly set track_selection when it'll only be operating on a single element | 14:15 |
| jonathanmaw | but I guess that's what comments are for | 14:15 |
| juergbi | yes, appropriate comments are always good | 14:15 |
| juergbi | to me it seems odd to use .ALL there in the first place despite only operating on a single element | 14:16 |
| juergbi | it just doesn't seem odd right now because it's the default value of the parameter | 14:16 |
| *** bochecha_ has quit IRC | 14:42 | |
| *** bochecha_ has joined #buildstream | 14:46 | |
| gitlab-br-bot | buildstream: merge request (372-allow-queues-to-run-auxilliary-jobs-after-an-element-s-job-finishes->master: WIP: Resolve "Allow queues to run auxilliary jobs after an element's job finishes") #433 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/433 | 14:50 |
| tristan | jonathanmaw, juergbi; not getting into the details but; please keep in mind that _pipeline.py is IMO a mess and needs separation, it being very confusing to follow is most probably a huge source of confusion here | 14:57 |
| tristan | selection of elements from the graph needs to be decoupled from the methods which execute the scheduler | 14:58 |
| juergbi | indeed. I'm trying to avoid complicating it further | 14:58 |
| tristan | yeah, I'm not getting into details but that is indeed the impression I'm getting | 14:59 |
| tristan | :) | 14:59 |
| tristan | juergbi, thanks :) | 14:59 |
| tristan | we have a bit of a crooked foundation there, lets try not to "build on it" | 14:59 |
| juergbi | yes, I think with the latest plan we will no longer pile further on top, even though we also don't do any untangling of existing stuff as part of this MR | 15:01 |
| tristan | good idea :) | 15:02 |
| tristan | I have some notes and a bit of a start towards a refactor | 15:02 |
| tristan | I started by just listing the operations we do for lists of elements | 15:02 |
| juergbi | great, maybe we should put this in an issue or so | 15:03 |
| tristan | If I can split that into a separate thing, then maybe I can get a higher level caller to make the decision of what to process, and feed that into existing codepaths | 15:03 |
| tristan | Yeah, we could issue it | 15:04 |
| * tristan is in off mode for the moment | 15:04 | |
| tristan | but it's in my foreground processes anyway, I wont forget without an issue | 15:04 |
| juergbi | ok | 15:04 |
| tristan | interesting new thread on buildstream-list, too | 15:04 |
| juergbi | I hope we'll also get to moving away from fork at some point | 15:05 |
| juergbi | package managers? | 15:05 |
| tristan | yeah, I wouldnt necessarily named it as such | 15:05 |
| tristan | but it's an important unsolved problem | 15:05 |
| tristan | pip,npm,cargo,etc | 15:06 |
| juergbi | yes | 15:06 |
| *** xjuan has joined #buildstream | 15:19 | |
| gitlab-br-bot | buildstream: merge request (jennis/136-clean-remote-cache->master: WIP: jennis/136 clean remote cache) #421 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/421 | 15:32 |
| *** xjuan has quit IRC | 15:39 | |
| milloni | tristan, i'll be known as milloni :) | 15:43 |
| gitlab-br-bot | buildstream: merge request (jennis/136-clean-remote-cache->master: WIP: jennis/136 clean remote cache) #421 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/421 | 15:49 |
| gitlab-br-bot | buildstream: merge request (jennis/136-clean-remote-cache->master: WIP: jennis/136 clean remote cache) #421 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/421 | 15:54 |
| gitlab-br-bot | buildstream: merge request (jennis/136-clean-remote-cache->master: WIP: jennis/136 clean remote cache) #421 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/421 | 15:56 |
| gitlab-br-bot | buildstream: merge request (jennis/136-clean-remote-cache->master: WIP: jennis/136 clean remote cache) #421 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/421 | 15:57 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 16:23 |
| gitlab-br-bot | buildstream: merge request (372-allow-queues-to-run-auxilliary-jobs-after-an-element-s-job-finishes->master: WIP: Resolve "Allow queues to run auxilliary jobs after an element's job finishes") #433 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/433 | 16:24 |
| *** xjuan has joined #buildstream | 16:29 | |
| gitlab-br-bot | buildstream: merge request (372-allow-queues-to-run-auxilliary-jobs-after-an-element-s-job-finishes->master: WIP: Resolve "Allow queues to run auxilliary jobs after an element's job finishes") #433 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/433 | 16:30 |
| *** Prince781 has joined #buildstream | 16:31 | |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 16:31 |
| gitlab-br-bot | buildstream: merge request (jennis/136-clean-remote-cache->master: WIP: jennis/136 clean remote cache) #421 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/421 | 16:37 |
| gitlab-br-bot | buildstream: merge request (jennis/136-clean-remote-cache->master: WIP: jennis/136 clean remote cache) #421 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/421 | 16:38 |
| *** bethw has quit IRC | 16:49 | |
| *** jonathanmaw has quit IRC | 17:01 | |
| *** dominic has quit IRC | 17:02 | |
| *** toscalix has quit IRC | 17:03 | |
| *** toscalix has joined #buildstream | 17:08 | |
| *** toscalix has quit IRC | 17:09 | |
| *** Prince781 has quit IRC | 18:59 | |
| *** Prince781 has joined #buildstream | 19:02 | |
| *** Prince781 has quit IRC | 19:18 | |
| *** Prince781 has joined #buildstream | 19:23 | |
| *** Prince781 has quit IRC | 19:24 | |
| *** bochecha_ has quit IRC | 19:42 | |
| *** tristan has quit IRC | 19:58 | |
| *** Prince781 has joined #buildstream | 20:27 | |
| *** bethw has joined #buildstream | 20:42 | |
| *** bethw has quit IRC | 20:46 | |
| *** aday has quit IRC | 21:46 | |
| *** aday has joined #buildstream | 21:49 | |
| *** aday has quit IRC | 22:00 | |
| *** Prince781 has quit IRC | 22:00 | |
| gitlab-br-bot | buildstream: merge request (378-make-bst-here-more-flexible->master: WIP: Resolve "Make `bst-here` more flexible") #439 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/439 | 22:41 |
| *** xjuan has quit IRC | 22:47 | |
| gitlab-br-bot | buildstream: merge request (378-make-bst-here-more-flexible->master: WIP: Resolve "Make `bst-here` more flexible") #439 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/439 | 23:37 |
| gitlab-br-bot | buildstream: merge request (378-make-bst-here-more-flexible->master: Resolve "Make `bst-here` more flexible") #439 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/439 | 23:40 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!