| *** semanticdesign has quit IRC | 00:18 | |
| *** cs_shadow has quit IRC | 03:43 | |
| *** bochecha has joined #buildstream | 04:26 | |
| *** bochecha has quit IRC | 05:37 | |
| *** tristan has joined #buildstream | 06:18 | |
| *** givascu has joined #buildstream | 09:03 | |
| *** givascu has quit IRC | 09:10 | |
| *** givascu has joined #buildstream | 09:10 | |
| *** tiagogomes has joined #buildstream | 09:17 | |
| *** jonathanmaw has joined #buildstream | 09:18 | |
| *** givascu has quit IRC | 09:22 | |
| *** Ebardie has joined #buildstream | 09:24 | |
| *** givascu has joined #buildstream | 09:39 | |
| *** ssam2 has joined #buildstream | 10:06 | |
| *** valentind has joined #buildstream | 10:09 | |
| tlater | tristan: Have you seen sanders' response to my mail? | 10:38 |
|---|---|---|
| tristan | tlater, havent read it yet, replied to various other emails and bug reports today | 10:39 |
| tristan | tlater, will be looking at it presently | 10:40 |
| valentind | It seems integration commands on compose cannot delete files. Or replace a directory or a file by a symbolic link. | 10:41 |
| valentind | It would be nice compose only staged in / and returned / as output directory. | 10:42 |
| tristan | valentind, I dont understand exactly precisely what you are saying, I need verbosity :-/ | 10:45 |
| valentind | I guest script with no commands and install-root set to / is exactly it. | 10:45 |
| tristan | A script element can be used to do a morph directly on / indeed | 10:45 |
| valentind | So I wrote an integration command that moved a directory and replaced by a symbolic link to the new place. | 10:45 |
| valentind | And I had a compose depending on the element with that integration command. | 10:46 |
| tristan | we use it do run `dpkg --configure -a` on a multistrap output for instance | 10:46 |
| valentind | The result was just a copy of the directory. | 10:46 |
| tristan | That *might* be a bug | 10:46 |
| valentind | And the problem is because compose stages twice dependencies. | 10:46 |
| tristan | symlinks are hard things - also we normalize them at stage time | 10:47 |
| valentind | And tries to detect changes on the first staging and apply it to the second inside %{install-root} | 10:47 |
| tristan | Ahhh | 10:47 |
| tristan | I see | 10:47 |
| tristan | Yes this is very tricky | 10:47 |
| valentind | But if we had compose to have %{install-root} as / we probably would not need to stage twice. | 10:48 |
| valentind | But I am not sure why we need to have %{install-root} to not be /. | 10:48 |
| tristan | Errrm, not sure of this | 10:48 |
| valentind | Because there is not much you can do with compose anyway. And the runtime is not used. | 10:48 |
| valentind | So we could put it anywhere. | 10:48 |
| tristan | The compose element handles a problem we had with other tooling which tries to produce slimmed down sysroot output | 10:49 |
| tristan | That problem is basically... | 10:49 |
| tristan | You have 2 competing interests here, one is you want to remove stuff that you dont want in your output | 10:49 |
| tristan | The other is that you want the output to be runnable / chrootable and possibly bootable, which requires integration commands | 10:50 |
| tristan | These are at odds with eachother, because removing things from the output, if done first; means that integration commands cant run | 10:50 |
| tristan | I.e. you've removed the tools which you need to run, in order to run those commands | 10:50 |
| valentind | But how can you do this with compose? | 10:51 |
| valentind | You meant remove the sysroot component from the path? | 10:51 |
| tristan | There is no order of "split before running commands" or "run commands before split" which works, running the commands before split means you no longer know which files to capture using regex patterns or globs (there was more to this, but I dont recall every single thing) | 10:51 |
| valentind | There does not seem to be any configuration to be able to do that. | 10:51 |
| tristan | valentind, this is the reason why we stage twice | 10:52 |
| tristan | valentind, if we asked for integration, first we stage everything, no split rules apply | 10:52 |
| valentind | Ah, strip rule? | 10:53 |
| valentind | OK | 10:53 |
| tristan | valentind, at which point we snapshot the staged sysroot before and after running integration commands | 10:53 |
| tristan | if we asked to include integration, the files which were effected, are included in the result, after staging the split portions of the artifacts we want to include, into %{install-root} | 10:53 |
| tristan | so updated caches and newly created files which dont exist in any artifact (are created by integration commands), squash whatever is in the compose output | 10:54 |
| *** Beth has joined #buildstream | 10:55 | |
| tristan | "split rule" is ... http://buildstream.gitlab.io/buildstream/public.html#split-rules | 10:55 |
| valentind | Can we give the result of split rules to utils.move_files? | 10:55 |
| valentind | And not restage | 10:55 |
| tristan | Perhaps | 10:55 |
| valentind | Then that would solve my problem. | 10:56 |
| tristan | Certainly not as a split second decision | 10:56 |
| tristan | valentind, can you open a bug and outline the problem and proposed solution so we can take a bit of time to think about it ? | 10:56 |
| valentind | OK, I will do that this evening. | 10:57 |
| tristan | it sounds to me like it's a sane approach | 10:57 |
| valentind | I will also try it. | 10:57 |
| valentind | Maybe make an example with split rules. | 10:57 |
| tristan | but it's also delicate :) | 10:57 |
| tristan | it will also maybe require some changes in how we do things; currently I think we have Element.stage() which accepts split domains, maybe we need Element.manifest() which accepts split rules and lists the files which match the splits as an intermediate function used by Element.stage() | 10:58 |
| tristan | s/maybe we need Element.manifest() which accepts split rules/maybe we need Element.manifest() which accepts split domains/ | 11:00 |
| *** Beth has quit IRC | 11:00 | |
| *** bethwhite has joined #buildstream | 11:02 | |
| valentind | Ah I see, the split rules apply to the dependency. For example if you say "bin/*" it is only for the files of the dependency. | 11:06 |
| tristan | exactly | 11:06 |
| tlater | tristan: The except algorithms are both going to be a lot more efficient if we keep track of the parents of an element, can I talk you into allowing that? | 11:06 |
| tristan | elements themselves can also override/modify the rules for themselves | 11:06 |
| tristan | deriving from the project defaults | 11:07 |
| valentind | New files from integration commands have to obey the split rules of the same dependency, I suppose. | 11:08 |
| tristan | tlater, as in, both approaches (choice in how to treat the "purple elements") are improved by storing reverse dependencies ? | 11:08 |
| tristan | valentind, no - integration commands are not captured by split rules | 11:09 |
| tristan | valentind, which is why the compose element either does it, or doesnt do it | 11:09 |
| valentind | Ah ok | 11:09 |
| tlater | tristan: Yes, because in either case elements will have to be removed from their parents' dependencies | 11:09 |
| valentind | Then it is simpler. | 11:09 |
| tristan | tlater, I dont follow | 11:10 |
| valentind | We still have to track which files have been modified to remove them from the split rule of the dependency the original file depended on. | 11:10 |
| tristan | tlater, we're not modifying the build graph here, we're masking a list of which elements we're going to process | 11:11 |
| tristan | tlater, if we've excepted things - the build graph still has the dependencies, only the list we feed to the scheduler has changed | 11:11 |
| tristan | valentind, we have to track what files the integration commands modified (or created), yes | 11:12 |
| tristan | valentind, whether we have to consider which artifact they are related to, or which element's integration commands effected it, I dont think is relevant | 11:12 |
| tristan | valentind, in any case; we just copy (or move) them over *after* preparing the rest of the result | 11:13 |
| tlater | Yeah, you're right, that's still possible. | 11:16 |
| tlater | o\ | 11:16 |
| tristan | valentind, I think what you're getting at is basically "In a world where we move instead of re-stage at compose time, we need an utils.move_files() which explicitly ignores EEXIST errors on the source) | 11:16 |
| tristan | " | 11:16 |
| valentind | I will look at that later. Have to go. Thanks for the answers. | 11:32 |
| *** valentind has quit IRC | 11:32 | |
| tristan | tlater, so my understanding of sander's reply is that he would prefer to keep the purple element, right ? | 11:42 |
| tlater | tristan: Yup | 11:43 |
| tlater | Though I think the other part is more interesting | 11:43 |
| tlater | We managed to create an option for `bst build --track` that `bst track` doesn't have | 11:43 |
| tlater | I mean, there's --deps none, but it's not as flexible | 11:44 |
| * tristan is getting to that... | 11:45 | |
| tristan | whew, finally got that out | 12:10 |
| * tlater agrees with just having --track be recursive | 12:14 | |
| tristan | tlater, I'm gonna scram now... | 12:15 |
| tristan | yeah I think it was a good idea to raise this on the list | 12:15 |
| tristan | seeing at how explosive this is, how many options can ensue | 12:15 |
| tristan | tracking profiles will also be cooler | 12:16 |
| tristan | I guess this means --track OR --track-save (mutually exclusive options ?) --track-except (can be specified multiple times, ONLY allowed if either --track or --track-save have been specified) ? | 12:17 |
| tristan | tlater, want to consider that for me, the goal being having a minimal but convenient CLI API surface, and summarize on the thread ? | 12:18 |
| tlater | tristan: Sure... Though I think --track-save works best as flag that just turns saving on/off | 12:18 |
| tlater | Rather than having different functionality from --track | 12:18 |
| tristan | maybe you're right | 12:18 |
| tristan | yeah | 12:18 |
| tlater | It's the same number of options anyway | 12:19 |
| tristan | but less of a huge CLI invocation | 12:19 |
| tristan | but anyway | 12:19 |
| tlater | tristan: ciao :) | 12:19 |
| tristan | it's better as a flag, because it will also apply to --track-profile <profile_name> | 12:19 |
| tristan | tlater, thanks for handling it ! | 12:20 |
| tlater | :) | 12:20 |
| *** tristan has quit IRC | 12:23 | |
| ssam2 | is it normal for the test suite to take 16 minutes to run ? | 13:02 |
| gitlab-br-bot | push on buildstream@sam/compose-log-splits (by Sam Thursfield): 1 commit (last: Log details of artifact splitting when building 'compose' elements) https://gitlab.com/BuildStream/buildstream/commit/6559b2e728a52a8ed566d2976be3a18c1643a718 | 13:26 |
| gitlab-br-bot | push on buildstream@sam/compose-log-splits (by Sam Thursfield): 1 commit (last: Log details of artifact splitting when building 'compose' elements) https://gitlab.com/BuildStream/buildstream/commit/2f497ab7f18f0bfe475b267ed8b9fac067b8e0be | 13:28 |
| gitlab-br-bot | buildstream: merge request (sam/compose-log-splits->master: Log details of artifact splitting when building 'compose' elements) #140 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/140 | 13:28 |
| gitlab-br-bot | buildstream: merge request (sam/compose-log-splits->master: Log details of artifact splitting when building 'compose' elements) #140 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/140 | 13:29 |
| gitlab-br-bot | push on buildstream@sam/compose-log-splits (by Sam Thursfield): 1 commit (last: Log details of artifact splitting when building 'compose' elements) https://gitlab.com/BuildStream/buildstream/commit/93b7841bdee20d3daf4103985b5a4f1c199e8a81 | 13:29 |
| gitlab-br-bot | buildstream: merge request (sam/compose-log-splits->master: Log details of artifact splitting when building 'compose' elements) #140 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/140 | 13:29 |
| ssam2 | with that patch out of the way; time to look at why my split rules are bonkers | 13:34 |
| *** tlater has quit IRC | 13:36 | |
| *** laurenceurhegyi has quit IRC | 13:36 | |
| ssam2 | oh, they seem less bonkers now. perhaps i was printing something wrongly before | 13:36 |
| *** anahuelamo has quit IRC | 13:37 | |
| *** anahuelamo has joined #buildstream | 13:37 | |
| *** jonathanmaw has quit IRC | 13:37 | |
| *** jonathanmaw has joined #buildstream | 13:37 | |
| *** bethwhite has left #buildstream | 13:38 | |
| *** tpollard has quit IRC | 13:39 | |
| *** tiagogomes has quit IRC | 13:47 | |
| *** tiagogomes has joined #buildstream | 13:47 | |
| *** jonathanmaw has quit IRC | 13:48 | |
| *** tiagogomes has quit IRC | 13:48 | |
| *** tiagogomes has joined #buildstream | 13:48 | |
| *** jonathanmaw has joined #buildstream | 13:48 | |
| *** jonathanmaw has quit IRC | 13:51 | |
| *** jonathanmaw has joined #buildstream | 13:51 | |
| *** Ebardie has quit IRC | 13:51 | |
| *** tiagogomes has quit IRC | 13:52 | |
| *** tiagogomes has joined #buildstream | 13:52 | |
| *** laurenceurhegyi has joined #buildstream | 13:59 | |
| *** WSalmon has joined #buildstream | 14:00 | |
| *** tlater has joined #buildstream | 14:00 | |
| ssam2 | i have a gnu-toolchain.bst stack.. which contains glibc, gcc, gawk, m4, make, etc. | 14:10 |
| ssam2 | my final system contains m4, gawk and make, which is not what I want | 14:10 |
| ssam2 | but how do I remove those ... | 14:10 |
| ssam2 | do I put /usr/bin/make into the 'devel' artifact for make ? | 14:11 |
| ssam2 | do I put /usr/bin/gawk into the 'devel' artifact for gawk ? | 14:11 |
| ssam2 | but then what if there's a program which actually depends on gawk at runtime ? | 14:11 |
| ssam2 | i guess the alternative is to remove gawk from the gnu-toolchain.bst stack | 14:13 |
| ssam2 | and force anything that requires gawk at runtime to actually depend on gawk | 14:13 |
| ssam2 | which seems a better way, really | 14:13 |
| *** tpollard has joined #buildstream | 14:42 | |
| ssam2 | i think I must have a bug here somewhere... sometimes the binaries from busybox end up in the 'devel' artifact | 14:50 |
| ssam2 | but no rule could be putting them there | 14:50 |
| ssam2 | and devel shouldn't be included in my artifact anyway! | 14:51 |
| ssam2 | maybe it's just being reported wrong | 14:52 |
| ssam2 | ah, my code to show domains was indeed wrong | 15:08 |
| *** jude has quit IRC | 15:20 | |
| *** jude has joined #buildstream | 15:36 | |
| *** cs_shadow has joined #buildstream | 15:58 | |
| gitlab-br-bot | push on buildstream@sam/compose-log-splits (by Sam Thursfield): 1 commit (last: Log details of artifact splitting when building 'compose' elements) https://gitlab.com/BuildStream/buildstream/commit/25974124bae37dd60db2bb43e9642abd3f1ad5ce | 16:04 |
| gitlab-br-bot | buildstream: merge request (sam/compose-log-splits->master: Log details of artifact splitting when building 'compose' elements) #140 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/140 | 16:04 |
| ssam2 | bah, another 15 minutes of building GCC wasted because of disk space being too low | 16:58 |
| *** tiagogomes has quit IRC | 17:04 | |
| *** valentind has joined #buildstream | 17:08 | |
| gitlab-br-bot | push on buildstream@97-apply-pep-3102-to-all-public-api-surfaces (by Jonathan Maw): 2 commits (last: Revert "Apply pep 3102 to Element.dependencies") https://gitlab.com/BuildStream/buildstream/commit/4347640c34059db55180920f9e953b00abc478ac | 17:33 |
| gitlab-br-bot | buildstream: merge request (97-apply-pep-3102-to-all-public-api-surfaces->master: WIP: Resolve "Apply pep 3102 to all public API surfaces") #127 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/127 | 17:33 |
| *** givascu has quit IRC | 17:46 | |
| *** WSalmon has quit IRC | 17:48 | |
| *** adds68 has quit IRC | 17:56 | |
| gitlab-br-bot | push on buildstream@sam/overwrite-list-with-empty-list (by Sam Thursfield): 1 commit (last: Allow overwriting a list with an empty list using (=) operator) https://gitlab.com/BuildStream/buildstream/commit/cb6df173ef00b28bdc5c0055fbd6f111d23217e9 | 18:01 |
| gitlab-br-bot | buildstream: merge request (sam/overwrite-list-with-empty-list->master: Allow overwriting a list with an empty list using (=) operator) #141 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/141 | 18:02 |
| * ssam2 deletes 51GB of BuildStream cache | 18:05 | |
| *** jonathanmaw has quit IRC | 18:05 | |
| ssam2 | oops, didn't realise he was in the cache | 18:06 |
| gitlab-br-bot | buildstream: merge request (sam/overwrite-list-with-empty-list->master: Allow overwriting a list with an empty list using (=) operator) #141 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/141 | 18:08 |
| gitlab-br-bot | push on buildstream@multiple_targets (by Tristan Maat): 4 commits (last: main.py: Fix app initialization) https://gitlab.com/BuildStream/buildstream/commit/b354b0f62e82b1d46193f1454c7d0d4c5bcaba28 | 18:08 |
| *** jude has quit IRC | 18:09 | |
| valentind | I get 49 errors running the tests. | 18:15 |
| valentind | Is that normal not all the tests pass? Or is it just on my computer? | 18:15 |
| ssam2 | all the tests should pass | 18:15 |
| ssam2 | you might be missing a dependency ? e.g. the bzr source tests require you to have bzr installed, I think | 18:16 |
| valentind | from buildstream import LoadError, LoadErrorReason | 18:16 |
| valentind | E ImportError: cannot import name 'LoadError' | 18:16 |
| valentind | This is what I get. | 18:16 |
| ssam2 | weird; what commit of buildstream.git ? | 18:16 |
| valentind | I just fetched master now. | 18:17 |
| valentind | 0f134712d3efb31cd1e3d53d464a8e66ee92a5f1 | 18:17 |
| ssam2 | the tests for that all passed in GitLab CI: https://gitlab.com/BuildStream/buildstream/commit/0f134712d3efb31cd1e3d53d464a8e66ee92a5f1/pipelines | 18:17 |
| ssam2 | perhaps you have extra files in the build tree, or __pycache__ messing things up ? | 18:17 |
| ssam2 | removing python cache can sometimes fix weird errors like that | 18:18 |
| valentind | Now it is working but I am not sure why. | 18:20 |
| valentind | I removed __pycache__ already before. | 18:20 |
| valentind | I also uninstalled it from the system. Not sure if it was causing problems. | 18:21 |
| ssam2 | hmm, possibly an old buildstream installation in PYTHONPATH could cause that | 18:21 |
| ssam2 | seems like a bug in the test runner if so, we should keep an eye on that | 18:22 |
| ssam2 | anyway, i'm glad it works now :-) | 18:22 |
| *** jude has joined #buildstream | 18:25 | |
| valentind | tests/frontend/pull.py::test_push_pull[user-config] seems to hang | 18:33 |
| ssam2 | could be the testutils.create_artifact_share() stuff ? | 18:37 |
| ssam2 | that doesn't seem to do much beyond calling ostree though | 18:37 |
| valentind | Restarted it and it seems to have gone through. | 18:38 |
| ssam2 | i seem to recall a test randomly failing in CI | 18:38 |
| ssam2 | i wonder if it was the same one | 18:38 |
| ssam2 | it was: https://gitlab.com/BuildStream/buildstream/issues/144 | 18:38 |
| *** ssam2 has quit IRC | 18:41 | |
| valentind | Now I have only one test failing: test_build[zip] | 18:50 |
| valentind | BUG: Message handling out of sync, unable to retrieve failure message for element import element at workspace-test-zip.bst [line 1 column 0] | 18:51 |
| valentind | I might have ctrl-c'ed it. | 19:04 |
| *** tristan has joined #buildstream | 19:05 | |
| valentind | Were pull and push commands recently added? I did not realize they existed before. | 19:05 |
| valentind | Finally all the tests passed. I installed all the test dependencies on the system. Maybe there is a new version of some of those dependencies that make the tests unstable. | 19:24 |
| valentind | tristan, I wanted to write tests for the deleted files bug with integration on compose. I do not find any test doing integration with compose. Is that normal? Is it because it requires a runtime? | 19:36 |
| tristan | valentind, integration-tests directory is the place for that sort of test | 20:16 |
| tristan | it used to be a separate repo, we're thinking of making it use pytest but currently it's just some shell scripts | 20:17 |
| tristan | indeed, it's because it requires a runtime + downloading external resources, the regular `./setup.py test` suite in contrast is self contained and doesnt take loads of time to complete | 20:18 |
| valentind | OK, but I have started working on a mock plugin that overrides integration. | 20:25 |
| valentind | Very simple. | 20:25 |
| *** tristan has quit IRC | 22:52 | |
| *** adds68 has joined #buildstream | 23:28 | |
| *** adds68 has quit IRC | 23:35 | |
| *** jude has quit IRC | 23:40 | |
| gitlab-br-bot | buildstream: merge request (compose-integration-delete->master: Handle removed files from integration in compose plugin) #142 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/142 | 23:55 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!