| *** xjuan has quit IRC | 00:52 | |
| *** Prince781 has quit IRC | 03:14 | |
| *** tristan has quit IRC | 06:01 | |
| *** tristan has joined #buildstream | 06:28 | |
| tristan | juergbi, not sure why... but tests/artifactcache/junctions.py::test_push_pull has a tendency to hang on my machine | 08:41 |
|---|---|---|
| tristan | I have a currently hanged state | 08:41 |
| juergbi | i've seen hanging tests after a failure sometimes | 08:42 |
| juergbi | however, i don't recall hanging tests when they succeed | 08:42 |
| tristan | all green | 08:42 |
| tristan | and it's really often *that* test | 08:42 |
| tlater | tristan: Nexus has seen similar issues on a different test, worth asking him how far he's into debugging that | 08:42 |
| juergbi | do you see a backtrace when you interrupt? | 08:42 |
| tristan | I dont have problems with other tests that I can recall, actually | 08:42 |
| * tristan interrupts | 08:42 | |
| tristan | other tests running now... but I should get a capture... | 08:43 |
| tristan | tests/frontend/buildcheckout.py::test_build_checkout[strict-copies] failed | 08:43 |
| tristan | juergbi, no stack trace :-( | 08:48 |
| tristan | juergbi, https://bpaste.net/show/70eae6fb84d6 | 08:48 |
| tristan | that is junctions.py:test_push_pull() | 08:48 |
| tristan | and the other failing test, *does* have stack track | 08:49 |
| tristan | perhaps it can help inform about state during the run | 08:49 |
| * tristan pastes that | 08:49 | |
| tristan | juergbi, the other failing test, test_build_checkout[strict-copies]: https://bpaste.net/show/67681b54c980 | 08:50 |
| tristan | self.scheduler.loop is None for some reason | 08:51 |
| juergbi | so this appears to have hanged: | 08:52 |
| juergbi | [--:--:--][2613eb12][ push:target.bst ] START Preparing compressed archive | 08:52 |
| *** aday has joined #buildstream | 08:52 | |
| tristan | ostree strikes back ! | 08:52 |
| juergbi | do you use parallel execution of tests? | 08:52 |
| tristan | nope | 08:53 |
| juergbi | preparing compressed performs a local ostree fetch into archive-z2 | 08:54 |
| tristan | yep | 08:54 |
| tristan | That happens inside a child task | 08:55 |
| tristan | It could be while creating the repo (ensure_repo), or while fetching into the new temporary repo | 08:55 |
| juergbi | maybe there is still some fork-related ostree issue? | 08:56 |
| * tristan wonders if ostree is getting used again in the main process for some reason | 08:56 | |
| tristan | Or, maybe we need to be more aggressive about never importing `gi` in the main process | 08:56 |
| juergbi | we actually do use it in the main process | 08:57 |
| juergbi | e.g., for .contains() | 08:57 |
| tristan | Hrmmm | 08:59 |
| tristan | good point that we do; that is broken | 08:59 |
| tristan | sigh :( | 09:00 |
| tristan | juergbi, I do get that you will again say fork() without execve() is bad, I understand that | 09:00 |
| tristan | I dont hold that opinion, but my opinion doesnt matter, because nobody cares | 09:01 |
| juergbi | hehe, i was trying not to say anything ;) | 09:01 |
| juergbi | it's not bad if you control all code that are used in the process | 09:01 |
| tristan | and because nobody cares, people will still keep firing up threads in libraries without clearly communicating that to API consumers | 09:01 |
| juergbi | with 3rd party libraries that's pretty much impossible | 09:01 |
| tristan | it's only impossible because people just dont care | 09:02 |
| tristan | and in glib 10 years ago, we didnt really help | 09:02 |
| tristan | with the choice to unconditionally initialize threads | 09:02 |
| juergbi | times have changed | 09:02 |
| tristan | and because of general lack of diligence "we cannot have nice things" happens | 09:02 |
| tristan | yes :( | 09:02 |
| juergbi | threads are not the only issue with fork, though | 09:04 |
| juergbi | open socket connections | 09:04 |
| tristan | Similar thing, though | 09:04 |
| *** jennis has joined #buildstream | 09:05 | |
| tristan | I mean "we had ways" really, as long as you knew what you were doing, and libraries performed predictably | 09:05 |
| * tristan recalls a special case he had in place for the X11 socket, so that he could *instantaneously* spawn the jukebox UI on multiple remote displays without wasting time parsing xml and re-building hierarchies | 09:06 | |
| juergbi | fork can be convenient in some cases but i don't consider it a very elegant solution | 09:07 |
| juergbi | it's mostly a historic relict | 09:07 |
| tristan | it has different properties, but I would do well to agree with you on this | 09:08 |
| juergbi | and the original purpose was not as variant for concurrency | 09:08 |
| juergbi | it was a way to deal with little system memory. and there was no copy on write | 09:08 |
| juergbi | hm, that's odd, now i see tests/frontend/push.py::test_push_after_pull hanging here as well.. | 09:10 |
| tristan | Anyway; we are not "doomed", in the short term we can put out these fires by placing thread-using things into subprocesses, but for longevity, I guess we have to eventually switch to a threading model | 09:10 |
| juergbi | yes, we'll have to come up with a sensible plan... without breaking API | 09:16 |
| tristan | I suspect that part wont be too hard, given the nature of what plugins are doing | 09:18 |
| tristan | we've also been pretty careful about what tools we give them access to | 09:18 |
| tristan | hmmm, second failure in a row of test_build_checkout() with stack trace | 09:19 |
| *** jonathanmaw has joined #buildstream | 09:20 | |
| tristan | juergbi, I'm still not 100% sure about the game plan for cross-junction tracking, API-wise; or... I maybe dont like the plan | 09:24 |
| tristan | juergbi, first thing I'm gonna do in the branch; is lay down the law that it's not allowed unless the toplevel project uses project.refs; where it's supported | 09:24 |
| juergbi | that definitely makes sense | 09:25 |
| tristan | so that is either (A) bail out with error that recursive tracking has been requested where impossible, or (B) filter out those elements and print a warning | 09:25 |
| tristan | I am thinking... regular consistent --except options can be employed... and besides this, maybe --except-junctions convenience switch | 09:26 |
| tristan | This would mean, with the error; one could recommend `--except-junctions`, and in that way communicate that a recursive, unbounded with --except track command really means to track recursively | 09:27 |
| tristan | If we make the default the other way around, and require that cross-junction tracking take an additional switch; this feels like a more confusing interface | 09:28 |
| juergbi | yes, not sure about the best approach | 09:35 |
| juergbi | in a way i feel like the deps approach is not ideal for selecting elements to track | 09:35 |
| *** toscalix has joined #buildstream | 09:36 | |
| juergbi | i would typically either want to track only the specified element(s) or track all elements in a project | 09:36 |
| juergbi | but although that would again not make sense in context of tracking as part of a build | 09:37 |
| tristan | build + track should not be any different | 09:38 |
| tristan | Consider `build + track` as simply a sort of scheduling optimization "because we can" | 09:38 |
| tristan | that makes things easier to think about | 09:38 |
| juergbi | yes, this consistency makes a lot of sense | 09:38 |
| juergbi | jennis: sorry for the comment spam. the branch looks mostly good. just a few minor history fixes to avoid the current situation of broken tests for intermediate commits | 09:39 |
| tristan | I typically want to track only the specified elements, *or* I want to select a part of the stack for tracking; I probably rarely ever want to track the *whole project* | 09:39 |
| tristan | (I may want to track the whole project; when the whole project is fairly small and does not use any junctions, and I may want to track only the toplevel project when junctions are involved) | 09:41 |
| jennis | Hi juergbi, thanks, I'm working through them now | 09:41 |
| juergbi | tristan: with the whole project i meant without subprojects. this is for the situation such as a distro (or freedesktop-sdk) that might want to do daily or weekly tracking of all elements | 09:42 |
| tristan | right | 09:42 |
| juergbi | can just use a toplevel target and deps all of course | 09:42 |
| juergbi | and except-junctions | 09:42 |
| tristan | juergbi, in the GNOME project for instance, we are probably going to have a middle ground area; where things that GNOME depend on are not acceptable in freedesktop-sdk | 09:43 |
| tristan | this will most probably live in the base/ subdir, identifiable by base.bst stack | 09:43 |
| tristan | I can see this as a regular pattern too, "things you additionally need, but dont directly develop" | 09:44 |
| juergbi | yes, makes sense | 09:44 |
| juergbi | and use except for that | 09:44 |
| tristan | exactly | 09:44 |
| juergbi | sounds sensible | 09:45 |
| juergbi | tristan: project.refs and embedded refs are not usable in combination, correct? | 09:47 |
| tristan | It looks like the major blocker to addressing of junctioned elements is the loader | 09:47 |
| tristan | juergbi, correct | 09:47 |
| juergbi | i'm wondering, would it make sense to make separate-source-refs be a path | 09:47 |
| juergbi | and allow overriding in bst command line | 09:47 |
| tristan | juergbi, I have a fat warning when we load the pipeline, which lists the inline refs which are going to be ignored | 09:48 |
| juergbi | that way you could have a commited project.refs but the user could have his own uncommitted one | 09:48 |
| juergbi | the loader can already address junctioned elements for dependencies. it just doesn't support a single string syntax | 09:48 |
| tristan | alternative project.refs is a possibility, i.e. "use these refs instead" | 09:48 |
| tristan | juergbi, does the loader scan all the .bst files for junctions in an initial pass and load them first ? | 09:49 |
| juergbi | no, it's on demand | 09:49 |
| juergbi | first time encountered and cached | 09:49 |
| juergbi | but that should also work for the single string syntax, afaict | 09:50 |
| tristan | Hmmm, actually it doesnt need that | 09:50 |
| tristan | yeah, since the single string syntax specifies the junction.bst element first, we would only need to first load *that* one | 09:50 |
| tristan | invoking `bst -C sdk/ track bootstrap-junction.bst:diffutils.bst` ... well you can imagine from there what the loader has to do hehe | 09:51 |
| tristan | juergbi, does Plugin._get_full_name() work for deeply nested junctions ? | 09:57 |
| * tristan would expect something recursive there, maybe calling Plugin._get_full_name() on `project._junction`, instead of `project._junction.name` directly | 09:58 | |
| juergbi | tristan: as it is right now we merge junctions by name | 10:00 |
| juergbi | so we don't need the recursive approach | 10:00 |
| juergbi | however, i actually want to propose an alternative to that as it's not suitable for all cases | 10:01 |
| tristan | just trying to understand the flow of code | 10:01 |
| tristan | Mkay, I will avoid this addressing | 10:01 |
| tristan | for this branch | 10:02 |
| Nexus | was i mentioned? | 10:09 |
| *** jennis_ has joined #buildstream | 10:13 | |
| *** tiago has quit IRC | 10:16 | |
| *** tiago has joined #buildstream | 10:19 | |
| *** dominic has joined #buildstream | 10:20 | |
| tlater | Nexus: tristan found another hanging test case, probably worth reading the backlog and seeing if you have the same issue | 10:24 |
| Nexus | different test | 10:24 |
| tlater | In this case it was ostree being thread-unsafe, I struggle to see how this would affect your case, but worth looking into | 10:24 |
| juergbi | please note that the ostree thread issue is just a suspicion | 10:26 |
| juergbi | ostree is innocent unless proven guilty ;) | 10:26 |
| tristan | tlater, it is just a suspicion, and "thread-unsafe" is a very inaccurate description | 10:26 |
| juergbi | rather fork-unsafe | 10:27 |
| * tlater likes fork-unsafe :) | 10:27 | |
| tristan | which is fair - tlater rather "calling into ostree" from the main process, when we *know* we're going to later fork-without-execve, has been known to cause problems | 10:27 |
| Nexus | tristan: given you were quite involved in what i did in !310, could you have a look at what might be causing my test to hang please? I spent all of yesterday on it and couldnt resolve it. I got as far as it dying on `capture.startcapturing()` | 10:28 |
| juergbi | Nexus: i don't remember, is it hanging locally as well or only on gitlab? | 10:29 |
| Nexus | WHEN i can get it to run on my machine, and use BST_FORCE_BACKEND=unix, it fails locally too | 10:29 |
| *** tristan has quit IRC | 10:31 | |
| *** tristan has joined #buildstream | 10:33 | |
| tristan | Nexus, what is the test | 10:39 |
| tristan | name | 10:39 |
| tristan | I can't easily search through that pipeline for "failed" | 10:40 |
| Nexus | test_script_cwd | 10:40 |
| Nexus | juergbi: re: your comment on !305. If i don't have the contents of the package in the buildstream git, how can i test against it? | 10:47 |
| tristan | Nexus, ok so; that is a place where we are assigning %{build-root} in the element, you got that far right ? | 10:47 |
| Nexus | tristan: where sorrt? | 10:49 |
| Nexus | sorry* | 10:49 |
| tristan | tests/integration/project/elements/script/script-layout.bst | 10:50 |
| tristan | is *setting* %{build-root} | 10:50 |
| Nexus | right ok | 10:51 |
| tristan | This looks like it may be near the root cause of failures | 10:51 |
| Nexus | so do i need to change those values? | 10:51 |
| tristan | I dont have the answer at a glance | 10:52 |
| tristan | I think before changing anything, you need to figure it out | 10:52 |
| Nexus | i never even looked in that file | 10:52 |
| tristan | *why* the values would have to be changed, and *why* the test would be expected to pass or fail | 10:53 |
| tristan | Nexus, its a part of the test case which is failing | 10:53 |
| tristan | element_name = 'script/script-layout.bst' | 10:54 |
| Nexus | but the test: test_script_layout passes | 10:54 |
| tristan | jonathanmaw and I cooked up the layout API for script elements, which allows powerful manipulation of the mount mappings for generic script elements | 10:54 |
| Nexus | ah ok | 10:54 |
| tristan | There was not anything requiring %{build-root} about it | 10:55 |
| tristan | but it for some reason ended up in docs, I think we discussed removing this aspect between you and I | 10:55 |
| tristan | But before removing it from the test, it's better to first identify *if* and then *why* the modification of %{build-root} in script-layout.bst is effecting the outcome | 10:56 |
| Nexus | well won't it be redefined in later on? before build-root wasn't beign set was it? | 10:56 |
| tristan | Nexus, We may decide to forbid modifications to %{build-root} separately, that is not a reason to not discover why it might be messing things up | 10:57 |
| Nexus | What i mean is, %{build-root} is being set in the pipeline to /buildstream/%{project-name}/%{element-name} won't that effect this definition? | 10:59 |
| tristan | Nexus, iirc, this *works* on linux and works on your laptop but only fails on gitlab, in the unix case | 11:00 |
| Nexus | yes, unless i force it to run unix tests on my laptop | 11:00 |
| Nexus | which i unreliable at best | 11:00 |
| Nexus | is* | 11:00 |
| tristan | It fails with unix tests on your laptop ? | 11:00 |
| Nexus | ye | 11:00 |
| Nexus | s | 11:00 |
| tristan | Nexus, it's no different than on gitlab | 11:00 |
| tristan | Nexus, the FORCE_BACKEND thing exists *because* we dont have a real non-linux testbed | 11:01 |
| Nexus | it is since i have to manually remove files on my local machien to get them to run more than once | 11:01 |
| tristan | it's *also* what we use on gitlab | 11:01 |
| tristan | You mean host tools ? | 11:01 |
| tristan | or, unix tests are violating the rules ? | 11:02 |
| Nexus | no, i mean, i norder to run them as unix tests, i reed to run them as root, which mounts things | 11:02 |
| tristan | i.e. unix tests are still not idempotent ? | 11:02 |
| tristan | tlater, ^^^ is that the case ? | 11:02 |
| tlater | tristan: I was pretty sure they were | 11:03 |
| tristan | if so, we need to reopen or refile the test issue | 11:03 |
| tristan | Ok. | 11:03 |
| tlater | This is a bug if not | 11:03 |
| tristan | Nexus, I think I have an *idea* that something is wrong with the patch | 11:03 |
| tristan | tlater, ok so there is a bug; or something not entirely understood | 11:05 |
| tristan | oops s/tlater/Nexus | 11:05 |
| *** valentind has quit IRC | 11:06 | |
| tristan | Nexus, but I dont think this is what is causing the failure - because it passes on linux but not on fallback platform | 11:06 |
| tristan | Nexus, In your patch you assign 'build-root' variable in __init__, before instantiating Variables() | 11:06 |
| tristan | Nexus, which means that we *ignore* any modifications that are set in the element yaml, we overwrite everything. | 11:07 |
| tristan | Nexus, this is probably correct, and we can probably forbid any manual setting of `build-root` entirely, seems cleaner | 11:08 |
| tristan | Nexus, but as we dont forbid it; the assignment technically belongs in Element.__extract_variables(), *in between* the two calls to _yaml.composite() | 11:08 |
| tristan | or, before both | 11:09 |
| tristan | yeah before both, directly after _yaml.node_chain_copy() | 11:09 |
| tristan | Nexus, it will be interesting to know if the test starts to pass after changing that - if it does; then it's uncovering something quite wrong; which will require more detective work. | 11:10 |
| * juergbi is confused | 11:15 | |
| juergbi | tristan: aren't we still setting build-root via projectconfig.yaml by default? | 11:16 |
| juergbi | i don't see anything in __init__ in the latest patch | 11:16 |
| juergbi | also, i thought we agreed on %{element-name}, not %{element} | 11:16 |
| Nexus | tristan: do you mean `variables['element'] = self.name` | 11:18 |
| Nexus | that's the only thing i set in __init__ | 11:18 |
| tristan | juergbi, Yeah we agreed, element-name, not element; we're talking about build-root, the patch is about making build-root named by element | 11:22 |
| tristan | oh wait | 11:22 |
| tristan | juergbi, yeah you're right; I'm balony | 11:23 |
| tristan | sigh, I thought I had a hunch there | 11:23 |
| tristan | Nexus, I was talking about that; but I was wrong :-/ | 11:24 |
| tristan | Nexus, that is the correct place, no need to change that; build-root should be setup the same | 11:24 |
| Nexus | :( | 11:26 |
| tristan | Nexus, you are able to reproduce this on your machine right ? | 11:30 |
| tristan | Nexus, that is first step | 11:31 |
| Nexus | tristan: yes, but i then need to spenda good half an hour undoing all of the stuff it spews into my machine | 11:31 |
| Nexus | currently have the test hanging now | 11:31 |
| tristan | All the stuff it spews into your machine, that sounds very, very worrisome; talk to tlater about that | 11:32 |
| Nexus | shall do | 11:32 |
| tristan | Nexus, the next step, is to modify tests/testutils/runcli.py to print the command it's going to run *before* running it | 11:32 |
| tristan | i.e. the `if self.verbose` block where it tells you what command ran, well you wont get that | 11:33 |
| Nexus | tristan: i know the EXACT command it freezes on, i just don't know where to go from there | 11:33 |
| tristan | Ok, so you can run it outside of ./setup.py test | 11:33 |
| tristan | Which means, now you have real power | 11:33 |
| tristan | You can now run strace and such. | 11:33 |
| Nexus | the issue is, it freezes on capture.start_capturing(), which "I THINK" means something went wrong before then | 11:34 |
| tristan | So the bug *only* happens within the fixture ? | 11:34 |
| tristan | The command behaves as expected when you try to run it without ./setup.py test ? | 11:34 |
| Nexus | not sure | 11:38 |
| Nexus | i don't know how it works well enough to run it out of the test | 11:40 |
| Nexus | it's called in invoke | 11:40 |
| tristan | That's what I was saying about modifying runcli.py | 11:42 |
| tristan | See where it prints the command that it runs, *when* it runs that; it will leave the system state as it was | 11:42 |
| tristan | Which means you can copy/paste the command it ran and run it yourself | 11:42 |
| Nexus | ah i see | 11:42 |
| tristan | Nexus, except that since this is a *hang*, it wont get printed, because currently it only prints *after* self.invoke() | 11:43 |
| tristan | it's possible to reconstruct the command without that; by at least looking at the temp dir that it uses, but that's just a bit less convenient (you have to figure out what exactly the command was, with the --config arg and all) | 11:44 |
| juergbi | Nexus: !305: as tlater mentioned you only need the filenames, you don't actually check the content, afaict. and you might even just need one or a few filenames per test. but committing the full file list should also be acceptable | 11:58 |
| Nexus | ah i see | 11:59 |
| Nexus | i didn't know it was only the names checked | 11:59 |
| Nexus | although that makes sense | 11:59 |
| gitlab-br-bot | buildstream: merge request (issue-89_unique_build_dirs->master: WIP: Made modification to generate unique subdirs for built elements) #310 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/310 | 12:04 |
| Nexus | juergbi: i'd still need to upload the .deb file | 12:12 |
| tlater | Nexus: that's fine I'd say, though you could use a sample project you made. | 12:13 |
| gitlab-br-bot | buildstream: merge request (issue-243_dpkg_import_source_plugin->master: WIP: Created DPKG Source plugin for Issue #10) #305 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/305 | 12:14 |
| Nexus | kk replaced with empty files | 12:15 |
| gitlab-br-bot | buildstream: merge request (jmac/remove-sequence-id->master: Remove sequence ID.) #309 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/309 | 12:15 |
| jennis_ | juergbi, re that simplifiable if statement | 12:18 |
| jennis_ | https://paste.gnome.org/pwt33sguh | 12:18 |
| jennis_ | Also, if other people could share there thoughts. Basically, changing it to the latter will stop the warning however it's probably not as clear what that bit of code actually does. So do you think it's better to just disable the warning here or change the code to the latter? | 12:19 |
| jennis_ | their* | 12:19 |
| gitlab-br-bot | buildstream: merge request (issue-89_unique_build_dirs->master: WIP: Made modification to generate unique subdirs for built elements) #310 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/310 | 12:20 |
| *** jennis has quit IRC | 12:31 | |
| *** jennis has joined #buildstream | 12:31 | |
| *** jennis_ has quit IRC | 12:37 | |
| gitlab-br-bot | buildstream: merge request (215-workspace-builds-might-not-rebuild-correctly-when-dependecies-are-updated->master: WIP: Resolve "Workspace builds might not rebuild correctly when dependecies are updated") #315 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/315 | 12:40 |
| gitlab-br-bot | buildstream: merge request (215-workspace-builds-might-not-rebuild-correctly-when-dependecies-are-updated->master: WIP: Resolve "Workspace builds might not rebuild correctly when dependecies are updated") #315 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/315 | 12:42 |
| gitlab-br-bot | buildstream: merge request (215-workspace-builds-might-not-rebuild-correctly-when-dependecies-are-updated->master: WIP: Resolve "Workspace builds might not rebuild correctly when dependecies are updated") #315 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/315 | 12:42 |
| jmac | Trying to get my head around composition again | 12:44 |
| jmac | In https://buildstream.gitlab.io/buildstream/formatintro.html#format-composition, what's the difference between (1) and (2)? | 12:44 |
| juergbi | Nexus: .deb file: can't we use snapshot.debian.org? | 12:47 |
| juergbi | i don't think we should import random .deb files into buildstream git | 12:47 |
| juergbi | as it's a small file, it might not be too bad, but it feels odd to me | 12:49 |
| juergbi | as i mentioned a while ago, the alternative would be to also create the .deb as part of the test case | 12:49 |
| juergbi | if everyone else prefers importing the .deb, i won't argue over it | 12:50 |
| tlater | juergbi: I know this probably seems pointless, considering we're loading our base platform over the net, but it's probably better to avoid downloading from too many places in our test cases | 12:52 |
| juergbi | yes. however, snapshot.debian.org should guarantee availability, and we should be able to cache the source, i suppose. do we need to mark the test as integration test for this? | 12:53 |
| juergbi | that way it would only be downloaded once per system | 12:53 |
| juergbi | and it's a small .deb | 12:53 |
| tlater | Ah, you suggest we use buildstream to track the actual source? | 12:53 |
| juergbi | directly specify snapshot.debian.org in the url, yes | 12:54 |
| tlater | Hm, I don't think we need to mark it as an integration test for that, no | 12:54 |
| tlater | But it certainly would be nicer :) | 12:55 |
| tlater | Err, that probably is a bit ambiguous | 12:55 |
| tlater | We don't need to mark it as an integration test, I agree that it's a nicer solution | 12:55 |
| tlater | Although we wouldn't get the persistent cache | 12:56 |
| tlater | For a persistent cache we'd need to mark it as an integration test, yes | 12:56 |
| tlater | Also use the integration cli helper | 12:57 |
| *** tristan has quit IRC | 12:57 | |
| *** xjuan has joined #buildstream | 12:59 | |
| *** tristan has joined #buildstream | 13:04 | |
| tristan | jmac, I replied on the issue; but from the core perspective they can be considered the same; they are _yaml.composite_dict()'ed together | 13:31 |
| tristan | early on | 13:31 |
| jmac | Cool, it's starting to make sense | 13:32 |
| tristan | jmac, this works pretty much in the same way as data/userconfig.yaml and the user's optional ~/.config/buildstream.conf | 13:32 |
| jmac | Any idea about the thing I posted up there ^? About steps 1 and 2 looking similar? | 13:32 |
| jmac | Steps 1 and 2 on https://buildstream.gitlab.io/buildstream/formatintro.html#format-composition | 13:33 |
| Nexus | juergbi: wouldn't it be better to not require the internet to run tests? | 13:33 |
| tristan | jmac, the confusing part is that we have some optimization; there can be many element declarations (i.e. element.bst files) which are of the same *type* (or "kind") | 13:33 |
| tristan | jmac, so what we do; is the first time we resolve an autotools element for example; we hold on to that composition, in the Element.__defaults variable | 13:34 |
| juergbi | Nexus: for non-integration tests that's definitely an argument. for integration tests we need it anyway | 13:34 |
| tristan | those become the "defaults of the class", which only then need the "defaults of the instance" to be composited when elements get instantiated | 13:34 |
| juergbi | Nexus: as i mentioned, importing a tiny .deb should probably be fine, assuming there is no license issue. i would prefer generating the .deb on the fly but i won't argue about this | 13:35 |
| jmac | tristan: That just sounds like cacheing the results of the first few stages of composition | 13:35 |
| tristan | jmac, (1) is buildstream's builtin `data/projectconfig.yaml`... (2) is the project.conf of a project | 13:36 |
| tristan | for your above question | 13:36 |
| tristan | jmac, and yes; that is exactly it; it's caching at the type level to avoid re-composition for every instance | 13:36 |
| jmac | But (1) explicitly links to "Project configuration" and says it should be named project.conf | 13:36 |
| tristan | That could be a problem in the docs :) | 13:37 |
| jmac | Right | 13:37 |
| tristan | they need to explain better that they are of the same format | 13:37 |
| jmac | In (0) Hard-coded means in Python then, not in projectconfig.yaml | 13:37 |
| tristan | Oh gee | 13:38 |
| tristan | I dont like that (0) exists | 13:38 |
| jmac | The user shouldn't see the distinction between (0) and (1) anyway | 13:39 |
| tristan | jmac, that is dirty and should be removed; one of the things I very much liked when I initially drafted the docs and how they get generated; is inline including of the .yaml configuration where possible | 13:39 |
| jmac | OK, sounds like something I can fix | 13:39 |
| tristan | So yeah; if project.py is "defaulting stuff" hardcoded, that should be moved to projectconfig.yaml, such that it is accessible and viewable by the reader of documentation | 13:40 |
| tristan | and not an obscure undocumented value | 13:40 |
| tristan | Not all things have a default, though; e.g. project options; so there is a vector of things which cannot be self-documenting in the .yaml file itself | 13:42 |
| jmac | projectconfig.yaml is inlined in that document but it's right at the end | 13:44 |
| jmac | I'll put a link in to it | 13:44 |
| tristan | jmac, the first link should probably link directly to: https://buildstream.gitlab.io/buildstream/projectconf.html#builtin-defaults | 13:44 |
| * jmac nods | 13:44 | |
| tristan | and then second section could use a link to the toplevel doc I guess | 13:45 |
| gitlab-br-bot | buildstream: issue #298 ("Unix integration tests not unmounting on crash") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/298 | 13:53 |
| jmac | Any idea how to link to a particular paragraph in .rst files? The Sphinx/RST documentation has not helped so far. | 14:37 |
| gitlab-br-bot | buildstream: merge request (jmac/remove-sequence-id->master: Remove sequence ID.) #309 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/309 | 14:40 |
| jmac | n/m, got it | 14:43 |
| gitlab-br-bot | buildstream: merge request (jmac/composition-docs-fix->master: Docs: Better explanation of defaults) #316 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/316 | 14:53 |
| *** sstriker has joined #buildstream | 15:21 | |
| *** mcatanzaro has joined #buildstream | 15:32 | |
| Nexus | [tlater, jonathanmaw] \o/ | 16:25 |
| jmac | dominic: If there is a big difference between the times of two test runs, it'd be useful to see that, as it indicates a problem with our tests | 16:27 |
| * dominic nods | 16:27 | |
| jmac | I should get an example output format from the logfile scraper | 16:38 |
| *** dominic has quit IRC | 16:43 | |
| jennis | I've seen this cls used as an argument in many functions but can't find a class / import for it | 17:02 |
| jennis | Could anyone shed some light? | 17:02 |
| *** valentind has joined #buildstream | 17:07 | |
| jennis | Because I'm not sure where these functions cls.<function_name> get called | 17:15 |
| jennis | https://gitlab.com/BuildStream/buildstream/blob/master/setup.py#L165 | 17:15 |
| jennis | *how | 17:15 |
| jmac | I think it's part of the @classmethod decorator | 17:16 |
| jmac | Bit complicated because that definition is outside of a class. Later on, someone assigns ScriptWriter.get_args = get_args | 17:18 |
| jmac | So now you can call ScriptWriter.get_args(dist, header) and it should get 'ScriptWriter' in the cls argument | 17:18 |
| jmac | I *think* | 17:18 |
| *** sstriker has quit IRC | 17:18 | |
| jennis | right, thanks jmac | 17:24 |
| * jennis goes to readup on decorators | 17:25 | |
| aiden | jennis, decorators are just functions that take functions and return functions | 17:26 |
| jennis | ta aiden | 17:27 |
| *** dominic has joined #buildstream | 17:47 | |
| *** toscalix has quit IRC | 17:51 | |
| *** jonathanmaw has quit IRC | 18:05 | |
| *** slaf has quit IRC | 18:26 | |
| *** slaf has joined #buildstream | 18:26 | |
| *** slaf has joined #buildstream | 18:27 | |
| gitlab-br-bot | buildstream: issue #299 ("Build sometimes hangs forever after "Unknown exception in SIGCHLD handler"") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/299 | 18:42 |
| jjardon[m] | Hi, do we have an issue for inacurate timing reports? I've just noticed this: https://gitlab.gnome.org/GNOME/gnome-build-meta/issues/23#note_80447 | 18:51 |
| gitlab-br-bot | buildstream: issue #300 ("Total build time for a given component doesnt seem to match the individual time of the different tasks") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/300 | 18:53 |
| juergbi | oh, that's really slow | 19:07 |
| juergbi | here adwaita-icon-theme build took 1min 49s in total, geoclue 20s | 19:12 |
| juergbi | we already have an issue for inaccurate total session time (not including initialization) but i don't think we had a report for inaccurate element time yet | 19:13 |
| juergbi | not everything is part of a timed subtask, though, so some leftover is expected | 19:14 |
| juergbi | however, it's usually a small difference | 19:14 |
| juergbi | i would definitely also look into why this is so slow in the first place. due to low on RAM and/or spinning rust? or is the worker simply overloaded? | 19:16 |
| *** dominic has quit IRC | 19:39 | |
| *** xjuan has quit IRC | 21:07 | |
| *** mcatanzaro has quit IRC | 22:10 | |
| *** aday has quit IRC | 22:11 | |
| *** aday has joined #buildstream | 22:12 | |
| *** aday has quit IRC | 22:35 | |
| *** valentind has quit IRC | 23:19 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!