*** valentind has quit IRC | 00:37 | |
*** xjuan_ has joined #buildstream | 01:57 | |
*** xjuan_ has quit IRC | 03:02 | |
*** mcatanzaro has quit IRC | 05:10 | |
*** bochecha has joined #buildstream | 08:41 | |
*** jude has joined #buildstream | 09:02 | |
*** tristan has quit IRC | 09:11 | |
*** jude has quit IRC | 09:19 | |
*** jude has joined #buildstream | 09:36 | |
*** bethw has joined #buildstream | 09:39 | |
*** tristan has joined #buildstream | 09:44 | |
*** bethw has quit IRC | 09:50 | |
*** jonathanmaw has joined #buildstream | 09:52 | |
*** valentind has joined #buildstream | 09:56 | |
* juergbi is wondering about a small junction syntax change. see mailing list | 09:58 | |
*** ssam2 has joined #buildstream | 10:08 | |
*** bethw has joined #buildstream | 10:15 | |
gitlab-br-bot | buildstream: merge request (tracking-changes->master: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 10:17 |
---|---|---|
gitlab-br-bot | buildstream: merge request (tracking-changes->master: Tracking changes) #119 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/119 | 10:18 |
tlater | tristan: I think that MR should be ready now :) | 10:18 |
*** bethw has quit IRC | 10:20 | |
* tlater ended up not changing how the planner works, because it's a massive headache to keep the sorting correct when things are done in two steps. | 10:20 | |
*** bethw has joined #buildstream | 10:21 | |
gitlab-br-bot | buildstream: merge request (sam/multiple-caches->master: WIP: multiple remote cache support) #166 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/166 | 10:22 |
*** ssam2 has quit IRC | 10:30 | |
*** ssam2 has joined #buildstream | 10:31 | |
gitlab-br-bot | buildstream: merge request (132-loading-external-plugins-works-without-explicit-requirement-in-project-conf->master: Resolve "Loading external plugins works without explicit requirement in project.conf") #125 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/125 | 10:36 |
gitlab-br-bot | buildstream: merge request (bst-here-permissions->master: bst-here: Mitigate permission issues) #169 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/169 | 10:44 |
jonathanmaw | tristan, can you review https://gitlab.com/BuildStream/buildstream/merge_requests/125 when you have time? | 10:47 |
gitlab-br-bot | buildstream: merge request (bst-here-permissions->master: bst-here: Mitigate permission issues) #169 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/169 | 10:52 |
*** valentind has quit IRC | 11:10 | |
ssam2 | does anyone have a clue about why adding a new plugin to bst-external would fail in this way ? | 11:25 |
ssam2 | oh wait, it's a new failure now | 11:26 |
ssam2 | ignore me, my error was missing __init__.py | 11:26 |
tlater | ssam2: Didn't that happen last week too? | 11:26 |
ssam2 | errors due to python's weird "package" concept ? probably | 11:27 |
*** tristan has quit IRC | 11:36 | |
*** tristan has joined #buildstream | 11:38 | |
*** tristan has quit IRC | 12:15 | |
gitlab-br-bot | buildstream: merge request (juerg/recursive-pipelines->master: WIP: Junctions and subprojects) #160 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/160 | 12:26 |
gitlab-br-bot | buildstream: merge request (114-give-better-warnings-on-overlaps->master: WIP: Resolve "Give better warnings on overlaps") #165 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/165 | 12:58 |
gitlab-br-bot | buildstream: merge request (114-give-better-warnings-on-overlaps->master: WIP: Resolve "Give better warnings on overlaps") #165 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/165 | 13:02 |
gitlab-br-bot | buildstream: merge request (bst-here-permissions->master: bst-here: Mitigate permission issues) #169 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/169 | 13:19 |
gitlab-br-bot | buildstream: merge request (114-give-better-warnings-on-overlaps->master: WIP: Resolve "Give better warnings on overlaps") #165 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/165 | 13:26 |
gitlab-br-bot | buildstream: merge request (114-give-better-warnings-on-overlaps->master: WIP: Resolve "Give better warnings on overlaps") #165 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/165 | 13:47 |
*** tristan has joined #buildstream | 14:11 | |
gitlab-br-bot | buildstream: merge request (114-give-better-warnings-on-overlaps->master: WIP: Resolve "Give better warnings on overlaps") #165 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/165 | 14:14 |
tlater | juergbi: https://irclogs.baserock.org/buildstream/%23buildstream.2017-11-21.log.html#t2017-11-21T11:02:10 | 14:17 |
tlater | My problem is that, in fact, _yaml.composite() does not receive a provenance *all* the time anymore. | 14:18 |
tlater | It doesn't because in this case: https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_options/optionpool.py#L230 | 14:19 |
tlater | Well, the provenance is handled outside of the node, because optionals aren't actual nodes | 14:20 |
tlater | So creating a stack of provenances doesn't really solve the problem :| | 14:21 |
tlater | I do have a branch that solves that particular issue, but I think the overall idea was to avoid this kind of situation in the future, not just prop the bug. | 14:21 |
juergbi | hm, i'm not exactly a yaml provenance expert ;) | 14:23 |
tlater | Heh, that's alright, just a bit stuck for how to solve this in the future :) | 14:24 |
tristan | oh that one, would be awesome to tolve that one | 14:24 |
tristan | tlater, _yaml.composite() *never* "receives" provenance | 14:25 |
tristan | tlater, _yaml.composite() however *always* receives a node | 14:26 |
tristan | for both the target and the source | 14:26 |
tlater | tristan: Not for the source, due to the bit of code I linked earlier | 14:26 |
tristan | and the provenance of the node, or particular members, can always be extracted | 14:26 |
tristan | hmmm did I miss something earlier ? | 14:27 |
tristan | I'm looking at https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_options/optionpool.py#L230 | 14:27 |
tlater | 'values' there is not a node | 14:27 |
tlater | 'value', sorr | 14:28 |
tristan | now is it not ? | 14:28 |
tristan | How | 14:28 |
tlater | I'm not sure why it isn't, it shows up as an array | 14:29 |
tlater | Maybe *that* is the issue, actually | 14:29 |
tristan | tlater, trying to think here, but I believe we only ever support conditionals in dictionaries | 14:29 |
tristan | and that simplifies things | 14:29 |
tlater | Ah, right | 14:29 |
tlater | It's because technically the user can define it in an array. | 14:30 |
tristan | so in a conditional statement... (?) is the special key for a list of conditionals | 14:30 |
juergbi | maybe because of the list() conversion? | 14:30 |
tristan | each conditional itself is a dictionary | 14:30 |
juergbi | (on line 222)? | 14:30 |
tristan | and the key is the conditional statement, the value is the dictionary to be composited into the dictionary containing the special (?) key | 14:30 |
* tlater looks a bit further into why this is an array | 14:31 | |
tristan | the list conversion there converts the generator into a list, I believe for the purpose of calling len() | 14:32 |
tristan | the generator yields a key/value tuple for each conditional | 14:33 |
tristan | which is the expression (key) and value to apply, which... is it always a dict ? | 14:33 |
tristan | I think it must always be a dict, since (?) is always a dict | 14:33 |
tlater | It isn't, oddly enough | 14:33 |
tristan | in what case ? | 14:33 |
* tlater finds the test case | 14:33 | |
tlater | It turns into an array containing this node: https://gitlab.com/BuildStream/buildstream/blob/8c4d8ca2b7c886690fb933428c6768c745667db7/tests/format/list-directive-type-error/element.bst#L6 | 14:34 |
tristan | maybe we need to assert that as an invariant | 14:34 |
tlater | (Actually the line above that) | 14:34 |
tristan | oh my | 14:34 |
tristan | that looks wrong | 14:34 |
tristan | ssam2, see that line above ^^^^ | 14:35 |
tristan | so... do we want to support that ? | 14:35 |
*** laurenceurhegyi has quit IRC | 14:36 | |
tlater | tristan: The context of that test case is that it used to throw a trace | 14:36 |
*** nexus has quit IRC | 14:36 | |
*** paulsherwood has quit IRC | 14:36 | |
*** paulsherwood has joined #buildstream | 14:36 | |
*** nexus has joined #buildstream | 14:36 | |
tlater | ssam2 fixed it by checking for that explicitly | 14:36 |
tristan | see, _yaml.composite() has an assertion immediately on the source, but not on the target | 14:37 |
tristan | which states that it must be something that has `.get()` (i.e. a mapping of some sort) | 14:37 |
*** laurenceurhegyi has joined #buildstream | 14:37 | |
tristan | Ummm tlater I'm more inclined to make that case invalid | 14:38 |
tristan | it's the easier thing at least | 14:38 |
tlater | tristan: It is currently, the assertion makes it invalid :) | 14:38 |
tristan | :-S | 14:38 |
tlater | Only issue is that we can now occasionally get a node without a provenance in _yaml.composite() | 14:38 |
tristan | define currently ? | 14:38 |
tlater | currently == current bst master | 14:39 |
tristan | tlater, if it's not a dictionary, it's not a node | 14:39 |
tlater | Right, so I think the issue is that we should probably check if it is a node before we give it to _yaml.composite(), and throw a LoadError before we ever enter that function? | 14:39 |
juergbi | so have a special error path for that? | 14:40 |
tristan | ah that assertion, ok so... the value should be asserted to be a dictionary in process_one_node(), in optionpool.py | 14:40 |
tristan | and the provenance used for that, I suppose would be _yaml.node_get_provenance(condition_node, condition_member) | 14:40 |
tristan | or I suppose the provenance collected in that loop will do fine | 14:41 |
tristan | juergbi, exactly, special error path for that | 14:41 |
tristan | before throwing invalid stuff at _yaml.composite() | 14:41 |
tlater | And then get rid of the assertion in _yaml.composite() - that would solve the issue, yeah :) | 14:41 |
jjardon[m] | Hi, buildstream seems to be stuck here in the "checking" phase: "Checking: 132/092" doing a strace I keep getting https://paste.gnome.org/pov6ck588 ; I can open that file and it seems ok; any idea what can go wrong? | 14:43 |
tristan | I dont think there is a need to get rid of the other assertion in _yaml.composite(), is that what sam added ? | 14:43 |
tlater | tristan: Yup, but it can occasionally print an error without a provenance | 14:43 |
tristan | tlater, is it what sam added in that previous patch ? | 14:44 |
tristan | is that what the Yep means ? | 14:44 |
tlater | Yes, it is | 14:44 |
tristan | Ok, then make it simply `assert` | 14:44 |
tristan | instead of removing completely | 14:44 |
tlater | Right, makes sense | 14:44 |
tristan | that way if it ever happens, we should get a BUG: message or stack trace | 14:45 |
juergbi | jjardon[m]: you could print the name of the element that is being checked, https://paste.gnome.org/pzwqxjgvr | 14:47 |
juergbi | (as it's not sure that it's related to the patch file) | 14:47 |
* tlater wonders if that assertion should be on both source and target | 14:54 | |
tlater | But then we're at the slippery slope of asserting all argument types | 14:54 |
jjardon[m] | juergbi: Checking: 132/092 graphite2.bst | 14:55 |
tristan | tlater, meh, can just remove it if that's the case | 14:55 |
tristan | tlater, seems to be a good assertion to keep in place, though | 14:55 |
jjardon[m] | but Im pretty sure it will go to the next element after a while: strace is spitting the same error for the same 5 files all the time | 14:55 |
tlater | Hm, I'll just keep it on only the source, in case we ever run into this again | 14:56 |
*** mcatanzaro has joined #buildstream | 14:58 | |
jjardon[m] | yup: "Checking: 132/095 libproxy.bst" | 15:03 |
juergbi | jjardon[m]: do you mean the ENOTTY error? | 15:12 |
jjardon[m] | juergbi: yes; well I think is actually not an error but bst is open and closing the same files all the time | 15:14 |
jjardon[m] | let me try to comment those patches from the .bst files | 15:14 |
gitlab-br-bot | buildstream: merge request (142-potentially-printing-provenance-more-than-once-in-loaderrors->master: Issue #142: Ensure we don't append provenances twice) #170 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/170 | 15:15 |
jjardon[m] | ok, no more problems with the files anymore; now is spitting: | 15:15 |
jjardon[m] | mmap(NULL, 262144, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f4fc0f17000 | 15:15 |
jjardon[m] | munmap(0x7f4fc0f17000, 262144) = 0 | 15:15 |
ssam2 | the patch files will be being touched each time it calculates a cache key | 15:15 |
juergbi | the TCGETS ioctl is probably just Python trying to figure it whether this is a special file | 15:15 |
ssam2 | because local/patch sources need to hash the files they import in order to calculate their cache key | 15:15 |
juergbi | that's just memory allocation | 15:15 |
* tristan is building GNOME converted to tarballs on a fairly slow internet connection... | 15:16 | |
juergbi | maybe there is an infinite loop somewhere, or something very slow | 15:16 |
ssam2 | although, if it's calculating cache keys a lot, maybe that's the slow down | 15:16 |
ssam2 | does every component in your pipeline have a 'ref' ? | 15:16 |
tristan | I've built 47 modules now, and still have a constant 18 tasks going on | 15:16 |
ssam2 | or are you running `build --track` ? | 15:16 |
tristan | 8 fetches, 8 pushes and 4 builds | 15:16 |
ssam2 | tristan: my laptop would hang for sure with that config ! | 15:16 |
ssam2 | 4 fetches + 2 builds hung it this morning | 15:17 |
tristan | ssam2, you need to get a developer laptop man ;-P | 15:17 |
tristan | grandma's browsing and email utility, just not enough | 15:17 |
ssam2 | this machine is battle hardened | 15:17 |
jjardon[m] | it can very well that some of the .bst files is broken: let me check | 15:18 |
ssam2 | note that it's not /broken/ to not have a ref field, but I have noticed hangs involving `build --track` before | 15:18 |
gitlab-br-bot | buildstream: merge request (juerg/recursive-pipelines->master: WIP: Junctions and subprojects) #160 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/160 | 15:18 |
tristan | jjardon[m], of course, broken bst files should not cause hangs, there is certainly a bug *somewhere* in here :) | 15:18 |
tristan | ssam2, fwiw, I noticed something today... if a low level component in a big pipeline has no ref... it can take a long time for the pipeline to report the error | 15:19 |
* jjardon[m] happy to find new bugs :) | 15:19 | |
tristan | this seems again part of this regression of overcalculation of cache keys | 15:19 |
tristan | we should just "know" that reverse deps cannot get cache keys when their deps cant | 15:20 |
tristan | all comes back to our messy all over the place handling of transient element state | 15:20 |
gitlab-br-bot | buildstream: merge request (128-status-ticker-fails-to-update-periodically-on-some-builds->master: WIP: Resolve "Status ticker fails to update periodically on some builds") #123 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/123 | 15:20 |
tristan | ssam2, and I thought there was a bug with `bst track --deps all` "hanging" in a certain situation, which I worked around by not printing the toplevel target at the end | 15:21 |
tristan | because the loop becomes super exponential, it hangs at 100% cpu for a long time, and eventually completes | 15:22 |
ssam2 | might have been thinking of that | 15:25 |
ssam2 | ah no, mine was https://gitlab.com/BuildStream/buildstream/issues/149 | 15:25 |
gitlab-br-bot | buildstream: merge request (tar-symlinks->master: Fix untar of symlinks. Only hardlinks are relative to top of archive and should be normalized.) #163 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/163 | 15:42 |
gitlab-br-bot | buildstream: merge request (sam/plugin-log->master: plugin.py: Add log() method) #168 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/168 | 15:46 |
gitlab-br-bot | buildstream: merge request (invoking_page_changes->master: WIP: Separated :show-nested: into individual calls) #157 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/157 | 15:49 |
jjardon[m] | ssam2: success! that was the problem | 15:55 |
jjardon[m] | some git sources where using track: but not ref: | 15:55 |
tristan | jjardon[m], it is expected to maintain ref-less projects | 15:56 |
tristan | i.e. it is still a valid project | 15:56 |
jjardon[m] | problem is that the track: pointed to a ref: instead to a branch | 15:57 |
tristan | just not a buildable one, it should report that it is not buildable quickly, though | 15:57 |
tristan | Ah, and doesnt that work ? | 15:57 |
tristan | I thought that worked | 15:57 |
tristan | not that it's really an intended use case | 15:57 |
tristan | but when you `bst track` it, I thought it did put the same sha in ref: | 15:57 |
jjardon[m] | tristan: this is the patch that make it work: https://paste.gnome.org/p5rkcgcgy | 15:58 |
gitlab-br-bot | buildstream: issue #164 ("Minimise overlaps by having overlaps raise exceptions unless configured not to") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/164 | 15:58 |
tristan | strange... ok so ssam2, jjardon[m]... can you work out what exactly the bug was and make sure it is filed, please ? | 15:59 |
* tristan is unsure at the moment | 15:59 | |
jjardon[m] | sure | 15:59 |
tristan | but it sounds like buildstream should have been telling you something quickly, and instead left you wondering what was going on | 15:59 |
gitlab-br-bot | buildstream: merge request (sam/plugin-log->master: plugin.py: Add log() method) #168 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/168 | 16:05 |
jjardon[m] | tristan: yup | 16:07 |
*** tristan has quit IRC | 16:12 | |
gitlab-br-bot | buildstream: merge request (sam/plugin-log->master: plugin.py: Add log() method) #168 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/168 | 16:28 |
*** tristan has joined #buildstream | 16:32 | |
gitlab-br-bot | buildstream: issue #165 ("bst gets confused and hangs forever if a git element have a `track:` parameter but not `ref:` (ie, you try to build before running bst track)") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/165 | 16:34 |
tristan | jjardon[m], thanks for filing that... that is indeed strange :-/ | 16:37 |
nexus | tristan: What project would you like me to use as an example piece? | 16:38 |
tristan | hrm | 16:41 |
jjardon[m] | tristan: np, thanks to you, ssam2 and juergbi for your help! | 16:42 |
tristan | nexus, I guess you need one ? I mean... I dont like to tie the docs to a specific project | 16:43 |
tristan | but maybe for now we can use the GNOME modulesets as an example | 16:43 |
tristan | nexus, it is also planned to give a similar guide, make examples for creating projects and using various constructs provided by the existing core set of plugins | 16:43 |
tristan | nexus, so I'm thinking, we may want to later change our user facing examples, to use projects which we ship for example purposes | 16:44 |
tristan | nexus, so I guess the best is, for now; create the user facing workflow docs using GNOME as an example, but keep in mind that we will later want to change the specific examples used to ones that we provide in another project authoring section | 16:45 |
tristan | so keeping that in mind mostly means, try to keep the actual text outside of the examples as project agnostic as possible ? | 16:45 |
juergbi | using sshd in tests doesn't exactly seem ideal. we have the issue that we have to find an unused TCP port, have to generate a key pair, and sshd does a permission check on every path component of the key, which may or may not be an issue in the test environment | 16:46 |
juergbi | i'm wondering whether we should rather exec pushreceive without ssh. we would still test the largest part that way | 16:46 |
ssam2 | would paramiko be easier ? | 16:46 |
juergbi | possibly | 16:47 |
ssam2 | or executing pushreceive another way ... the main issue with that is we have to have some ugly custom protocol that we only use in tests, right ? | 16:47 |
juergbi | yes | 16:48 |
juergbi | one other option could be to make local path use pushreceive | 16:48 |
juergbi | just without ssh | 16:49 |
ssam2 | oh, that would make sense since we only use that for testing anyway | 16:52 |
juergbi | yes, would actually reduce the number of code paths in some points | 16:53 |
tristan | nod, we discussed this the other day :) | 16:54 |
tristan | or similar | 16:54 |
tristan | fwiw, I dont object to paramoko really, it seems like it would be nice to have some drop in replacement for the ssh/sshd thing | 16:55 |
tristan | that would require the least test-conditional code | 16:55 |
tristan | and best coverage | 16:55 |
juergbi | well, if we used paramiko for client as well, we'd already not test the 'ssh' invocation line | 16:56 |
juergbi | the only additional missing coverage with local pushreceive execution would be ssh url parsing, afaict | 16:56 |
tristan | so it's not a drop in replacement I guess :-/ | 16:56 |
tristan | but still, whatever you find easy and good, will be better than the nothing we have :D | 16:57 |
juergbi | paramiko server with openssh client could be a possibility. would probably be slightly less painful than openssh client+server and slightly better code coverage | 16:57 |
juergbi | not sure whether it's worth it | 16:58 |
ssam2 | local file + pushreceive seems a simpler first step either way | 17:00 |
juergbi | i agree | 17:01 |
gitlab-br-bot | buildstream: merge request (142-potentially-printing-provenance-more-than-once-in-loaderrors->master: Issue #142: Ensure we don't append provenances twice) #170 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/170 | 17:05 |
tlater | This is where I wish my event hooks branch was merged, I could set up a script to shut down my PC after this build finishes ;P | 17:27 |
gitlab-br-bot | buildstream: merge request (sam/plugin-log->master: plugin.py: Add log() method) #168 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/168 | 17:29 |
*** jonathanmaw has quit IRC | 17:56 | |
juergbi | executing bst-artifact-receive more or less works | 17:57 |
juergbi | however, we don't get coverage for that part. that needs some more work, or we use fork without exec for it | 17:57 |
*** bethw has quit IRC | 18:01 | |
*** ssam2 has quit IRC | 18:05 | |
*** WSalmon has quit IRC | 18:05 | |
*** adds68 has quit IRC | 18:05 | |
*** jude has quit IRC | 18:06 | |
*** tpollard has quit IRC | 18:06 | |
*** ssam2 has joined #buildstream | 18:16 | |
*** ssam2 has quit IRC | 18:19 | |
*** jude has joined #buildstream | 18:53 | |
*** WSalmon has joined #buildstream | 18:58 | |
*** adds68 has joined #buildstream | 19:00 | |
*** adds68_ has joined #buildstream | 19:14 | |
*** adds68 has quit IRC | 19:15 | |
*** mcatanzaro has quit IRC | 19:15 | |
*** mcatanzaro has joined #buildstream | 19:16 | |
gitlab-br-bot | buildstream: merge request (sam/multiple-caches->master: WIP: multiple remote cache support) #166 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/166 | 19:17 |
*** valentind has joined #buildstream | 19:29 | |
*** tpollard has joined #buildstream | 20:52 | |
*** valentind has quit IRC | 22:27 | |
*** mcatanzaro has quit IRC | 23:29 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!