| *** Prince781 has joined #buildstream | 01:06 | |
| *** Prince781 has quit IRC | 01:18 | |
| *** jsgrant has joined #buildstream | 02:25 | |
| *** jsgrant has quit IRC | 02:29 | |
| *** jsgrant has joined #buildstream | 02:30 | |
| *** leopi has quit IRC | 02:42 | |
| *** leopi has joined #buildstream | 02:43 | |
| *** leopi has quit IRC | 02:47 | |
| *** leopi has joined #buildstream | 02:47 | |
| *** leopi has quit IRC | 02:52 | |
| *** leopi has joined #buildstream | 02:58 | |
| *** leopi has quit IRC | 03:43 | |
| *** jsgrant has quit IRC | 04:09 | |
| *** jsgrant has joined #buildstream | 04:10 | |
| *** leopi has joined #buildstream | 04:12 | |
| *** leopi has quit IRC | 05:05 | |
| *** slaf has quit IRC | 05:16 | |
| *** slaf has joined #buildstream | 05:18 | |
| *** slaf has joined #buildstream | 05:18 | |
| *** slaf has joined #buildstream | 05:19 | |
| *** slaf has joined #buildstream | 05:19 | |
| *** slaf has joined #buildstream | 05:19 | |
| *** slaf has joined #buildstream | 05:19 | |
| *** slaf has joined #buildstream | 05:20 | |
| *** slaf has joined #buildstream | 05:20 | |
| *** slaf has joined #buildstream | 05:20 | |
| *** slaf has joined #buildstream | 05:20 | |
| *** jsgrant has quit IRC | 05:24 | |
| *** jsgrant has joined #buildstream | 05:28 | |
| *** tristan has joined #buildstream | 05:30 | |
| *** noisecell has joined #buildstream | 05:49 | |
| *** leopi has joined #buildstream | 06:00 | |
| *** jsgrant has quit IRC | 06:04 | |
| *** jsgrant has joined #buildstream | 06:05 | |
| *** noisecell has quit IRC | 06:09 | |
| *** ernestask has joined #buildstream | 06:25 | |
| *** jsgrant has quit IRC | 06:27 | |
| *** noisecell has joined #buildstream | 07:37 | |
| gitlab-br-bot | buildstream: merge request (Move-_list_dir_contents-to-__init__->master: Move _list_dir_contents to __init__.py) #538 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/538 | 07:45 |
|---|---|---|
| *** coldtom has joined #buildstream | 07:53 | |
| gitlab-br-bot | buildstream: merge request (relative_workspaces->master: WIP: Patch for issue #191 support relative workspaces) #504 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/504 | 07:58 |
| gitlab-br-bot | buildstream: merge request (relative_workspaces->master: WIP: Patch for issue #191 support relative workspaces) #504 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/504 | 08:00 |
| gitlab-br-bot | buildstream: merge request (relative_workspaces->master: WIP: Patch for issue #191 support relative workspaces) #504 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/504 | 08:00 |
| *** leopi has quit IRC | 08:32 | |
| *** jennis has joined #buildstream | 08:33 | |
| tristan | juergbi, around ? | 08:33 |
| juergbi | yes | 08:34 |
| tristan | juergbi, are you ready to land CAS ? | 08:35 |
| *** jonathanmaw has joined #buildstream | 08:36 | |
| *** noisecell has quit IRC | 08:36 | |
| juergbi | tristan: almost there. one test case and the proto rename is missing | 08:37 |
| juergbi | should be done by the end of the day | 08:37 |
| tristan | Great | 08:37 |
| tristan | tlater, around ? | 08:38 |
| juergbi | there is one further point pending, which is removing the abstract ArtifactCache class. however, looking into that I think just merging the two classes is not ideal, so I'd like to think about this some more and thus merge without that | 08:39 |
| tlater | tristan: I am now | 08:40 |
| tristan | juergbi, hrmmm understood... but here's the thing: We want to land ideally our four targets very quickly in master before landing so as to open up 1.3.x development ASAP | 08:40 |
| *** bethw has joined #buildstream | 08:41 | |
| tristan | juergbi, ideally architectural changes which you make to that after branching `bst-1.2` should also be backported to `bst-1.2`, this is strictly in the interest of making it easy enough to backport any related bugfixes moving forward | 08:41 |
| tristan | juergbi, that said, I'm not particularly picky about that detail | 08:42 |
| *** noisecell has joined #buildstream | 08:42 | |
| *** jennis has quit IRC | 08:42 | |
| tristan | tlater, are you able to focus on the local cache expiry stuff ? | 08:43 |
| juergbi | yes, but I think we can manage backports in that area | 08:43 |
| *** jennis has joined #buildstream | 08:43 | |
| *** leopi has joined #buildstream | 08:43 | |
| juergbi | I prefer merging it as is and then backport the refactoring than refactor now and be unhappy with the structure later | 08:43 |
| tristan | tlater, or rather, could we go over what problems still exist with the branch together, and see how close we are to land it ? | 08:43 |
| tristan | juergbi, understood, yeah that's what I mean... if there are changes like that which land post branching, it would be best to backport those changes | 08:44 |
| juergbi | sure, I will take care of that | 08:44 |
| *** tiago has joined #buildstream | 08:45 | |
| tlater | tristan: I don't think I'll land it today, but we can go through it - I wanted to during guadec anyway, see if you're happy with the new new new approach ;) | 08:45 |
| tristan | tlater, yeah I didnt think so today either... I'm kinda hoping for tomorrow though, just lets see if I'm overly optimistic :) | 08:46 |
| *** jennis has quit IRC | 08:46 | |
| tristan | tlater, going over juergbi's mega CAS branch at GUADEC... I *think* that it will not significantly conflict with your branch | 08:47 |
| tristan | it may add some API to the artifact cache, though | 08:47 |
| tlater | Good, I'm getting pretty tired of rebasing this thing... | 08:47 |
| tristan | so there is the part about calculating size and such which I suppose may need some changes | 08:48 |
| tlater | The actual delete method will probably also change | 08:48 |
| tlater | But those are fairly simple changes :) | 08:49 |
| *** tiago has quit IRC | 08:49 | |
| tristan | Yeah... the greater ordeal; scheduling the job and all that, should really be quite unaffected | 08:49 |
| noisecell | tristan, tlater, is this part of https://gitlab.com/BuildStream/buildstream/issues/416 or something similar which is going to be re-architecture later? | 08:52 |
| *** tiago has joined #buildstream | 08:52 | |
| tlater | noisecell: It's completely separate from the artifact cache expiry, if that's what you're asking | 08:53 |
| tlater | The `bst artifact delete` function might use some API created in this branch, but it should otherwise be unrelated | 08:53 |
| tristan | tlater, I was about to just give it a run through, but it would probably make more sense to discuss the "new new new approach" at a high level | 08:56 |
| noisecell | tlater, ok, although, given that is is "artifact management", I would suggest to manage in the same class/pluging.... are you expiring by date and size, how this is going to be configurable? | 08:56 |
| tlater | noisecell: Currently expiration happens based on how recently an artifact has been used | 08:57 |
| tristan | noisecell, it is not really possible to encapsulate expiration in a class really, this needs to touch a lot of areas at once | 08:58 |
| noisecell | tristan, tlater, ok, you know the code better than me | 08:58 |
| tristan | and yeah, we want to expire based on artifact *usage* dates, the value you are interested in there is most probably a creation date | 08:59 |
| *** tpollard has joined #buildstream | 08:59 | |
| *** jennis has joined #buildstream | 08:59 | |
| tlater | tristan: Waiting for gitlab to load to have a look at my changes again, but essentially, the logic of when to schedule jobs has been moved to a new 'Resources' class | 09:00 |
| tlater | When I want to schedule a job, I first check with the resources class if everything the job wants is available/exclusive stuff is not in use | 09:01 |
| tlater | The Queues decide what the jobs need | 09:01 |
| tlater | And all the scheduler does in this regard anymore is ask whether it's allowed to launch a certain job | 09:01 |
| tlater | Oh, and it no has an API to launch cleanup jobs, which pull/build invoke after a build finishes. | 09:02 |
| *** jennis has quit IRC | 09:02 | |
| *** jennis has joined #buildstream | 09:02 | |
| tristan | Ah | 09:04 |
| tristan | tlater, ok so we dont launch a separate job anymore for the cleanup ? | 09:05 |
| tlater | We still do, but the scheduler now is in control of that | 09:05 |
| tlater | Just a minor thing so I don't have to duplicate this across pull/build, really. | 09:05 |
| *** jennis has quit IRC | 09:05 | |
| tristan | Ummm, ok I'm a bit confused what that means... are you saying that upon completion of a pull/build we inform the scheduler whether a cleanup is needed, or smth ? | 09:06 |
| tlater | We ask the scheduler to please make sure our cache isn't overflowing | 09:06 |
| tristan | ok, sounds reasonable... I guess the queue does this ? | 09:06 |
| tlater | Yes | 09:06 |
| tristan | good, I want to avoid elements calling into the scheduler and keep call paths happening in a single direction, makes things easier to reason about :) | 09:07 |
| tristan | asides from that, have you run any reasonably large builds against this change ? | 09:08 |
| tlater | Not yet | 09:08 |
| tristan | ok, as I can see the CI is not passing right now... is that a big deal or a minor detail ? | 09:09 |
| tlater | I think I forgot to commit a file... I certainly had it passing locally at some point, just trying to get it to fit in the history. | 09:09 |
| tristan | ok so lets throw up a branch right away on freedesktop-sdk for this | 09:10 |
| tlater | A separate branch? Wouldn't it be enough to just build freedesktop-sdk locally? | 09:10 |
| tristan | tlater, here's what I'll do right away... I'm going to branch your branch, and bump the BST_ARTIFACT_VERSION as a test | 09:10 |
| tristan | then I'll create a freedesktop-sdk branch which builds this test branch | 09:11 |
| tristan | (or builds *using* this test branch) | 09:11 |
| tristan | tlater, building locally would also be good yeah, but if I can kick that off right away we might find stuff earlier | 09:12 |
| tlater | Ok - I'll try to get the branch back in working shape for now. | 09:12 |
| tristan | yeah I think you may be missing an __init__.py in tests somewhere looking at the CI failure | 09:13 |
| tristan | it's one of those annoying import failures | 09:13 |
| tlater | I think I'm actually missing the whole resources file somehow | 09:13 |
| tristan | or maybe the resources.py class is actually missing yeah | 09:13 |
| tlater | Hm, damn, I was working on a different laptop | 09:15 |
| tlater | I might have to go grab that laptop... | 09:15 |
| tristan | tlater, ok so I guess I'll call off my test then :) | 09:17 |
| tristan | having that file would be a first step... | 09:17 |
| * tristan thought at first only the CI was failing, but without that file, nothing will work | 09:18 | |
| tlater | Yep, rather stupid mistake, sorry. It might bee in an old version of the branch, too, checking there first. | 09:19 |
| tlater | grr, gitlab, stop highlighting my syntax and show me the changes :| | 09:22 |
| tlater | Yes! It was in a commit from a day before I started mucking with my history! | 09:28 |
| tristan | valentind, so in one of our BoFs, you raised a point that you would prefer to not use project options to declare different ways to build something - I was a bit impatient to talk about that at the time, it's a subject that has been churned a lot | 09:36 |
| tristan | valentind, but it's better if you understand where we are coming from :) | 09:36 |
| valentind | tristan, I was thinking of making the VM as a separate project and select the systemd option for the SDK through the junction. | 09:37 |
| tristan | valentind, so anyway I'd like to shed some light on that if I can.... one thing which comes to mind is when I proposed "flavors" to solve this problem in baserock: https://listmaster.pepperfish.net/pipermail/baserock-dev-baserock.org/2015-November/013337.html | 09:38 |
| valentind | tristan, One of the case I wondered would be, what if you want to build a VM image with systemd that contains flatpak with a preseeded freedesktop image (without systemd) so that flatpak works out of the box. | 09:39 |
| tristan | valentind, in general one of the biggest pains we had, when trying to maintain a core OS + a genivi baseline system + a Qt5 platform + a GNOME system + an openstack cloud system.... | 09:39 |
| tristan | valentind, is that every time one of the higher level projects needed "something" to be done differently, a compile option on some common module... then every reverse dependency of that "chunk" (or "element" in bst-speak)... needed also to be duplicated | 09:40 |
| valentind | tristan, also I was thinking of using include feature for example. The point is always to have a reference implementation. But have the ability to tweak it. But we should always be clear what is the reference. | 09:40 |
| * paulsherwood is thinking, now tristan has mentioned baserock, we maybe need to establish ybd => baserock definitions => convert => bst files => bst as a CI/CD pipeline so we can compare results for a while | 09:41 | |
| tristan | valentind, basically, after seeing how intensely duplicated things can get, we rather want to encourage minimization, and not have to define an element twice for the same build; conditionals is intended to buy us this | 09:43 |
| tristan | valentind, that said; the original idea of "variants" had a concept of elements declaring what configuration they want from their dependencies; this API mitigates your concern that someone needs to know the options they want to use to build a given output; unfortunatly that approach didnt work because it's severely non-performant | 09:44 |
| tristan | So my suggestion is that, we try to explore ways to make this easier to follow; it could be that there is some middle ground which would allow a "target" to declare it's desired options, or have some project configuration to assist the user, I'm not sure | 09:45 |
| valentind | tristan, Gentoo has a similar feature. They call it "use". | 09:46 |
| tristan | valentind, this is something we very intentionally tried to reject | 09:46 |
| tiago | I was thinking in giving a shot at https://gitlab.com/BuildStream/buildstream/issues/263. Has anyone has opinions regarding whether this should be a separate subcommand or an option of `bst checkout` | 09:46 |
| tristan | valentind, the problem with the gentoo USE flags approach, is that it promotes the declaration of options on a module-by-module basis | 09:46 |
| *** rdale has joined #buildstream | 09:47 | |
| tiago | Also should the compression format be configurable? The most efficient compression format for Linux won't probably work out of the box in OSX and Windows | 09:47 |
| tristan | valentind, if you structure a project *that* way, then you have a combinatorial explosion of possible configurations which cannot all be tested (and which are not all interesting to deploy, either) | 09:47 |
| valentind | tristan, sure options should not be declared by elements. | 09:48 |
| tlater | tiago: I think ssssam[m]'s aim was to dump a tar stream to stdout which you can then do whatever you want with - no reason to implement different compression methods | 09:48 |
| tristan | valentind, so instead, we rather the project have a *few* options, based on which some modules may compile differently; allowing one to dramatically reduce the options to only the ones you are using | 09:48 |
| rdale | is there a way to purge the buildstream artifact cache of out of date artifacts? | 09:48 |
| tlater | It should probably be an option to checkout - though I wonder if we should just dump a tarfile instead. Printing to stdout strikes me as counterintuitive. | 09:49 |
| tristan | rdale, not yet; tlater is working on automatic expiry / local artifact cache quota... which we are trying to land in the next days | 09:49 |
| rdale | ok thanks no problem | 09:50 |
| tlater | rdale: Feel free to follow https://gitlab.com/buildstream/buildstream/merge_requests/347 for updates! | 09:50 |
| rdale | ok | 09:50 |
| tiago | What about if the directory argument is not given, bst checkout generates a tar stream. Maybe a bit magic though | 09:50 |
| tristan | rdale, a `bst artifact <foo> ...` commands for manipulating artifacts are also being thought of and discussed... around here: https://gitlab.com/BuildStream/buildstream/issues/416#note_83595827 | 09:50 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 09:51 |
| tlater | tiago: I like that better, but it doesn't mix well with what checkout currently does | 09:51 |
| ssssam[m] | i was imagining `bst checkout --tar` which would write a tar stream to stdout | 09:52 |
| gitlab-br-bot | buildstream: merge request (jmac/virtual_directories->master: WIP: Abstract directory class and filesystem-backed implementation) #445 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/445 | 09:52 |
| tlater | ssssam[m]: Any reason why you want it to be written to stdout? | 09:53 |
| tristan | tlater, I have a reason | 09:53 |
| tristan | tlater, basically, a tar stream can be passed to a compression tool or whatever the user might want before round tripping on disk and without growing more features on the BuildStream side, is one... | 09:54 |
| tristan | tlater, generally interoperability | 09:54 |
| tristan | One of the bigger arguments is that we'd really like to patch ostree to accept commits from stdin | 09:55 |
| tristan | tlater, ostree *already* accepts committing from a tarball | 09:55 |
| ssssam[m] | yeah, a tar stream can also be piped straight into `docker import` for example | 09:55 |
| tristan | this way we can do: bst checkout foo.bst --tar | ostree commit -m "This is the commit !" | 09:55 |
| tristan | right, or docker | 09:56 |
| tlater | Hrm, I dislike what it means for the UI | 09:56 |
| tlater | We already have a few people complaining about verbosity, who'd like the default to be checking out into a directory with the same name as the element | 09:56 |
| tristan | shouldnt need to scratch the disk for these things, and BuildStream CLI should not have to grow hard coded features for these things either | 09:56 |
| tiago | That isn't exactly a checkout of the artifact. Hence why I asked about separate subcommand | 09:56 |
| *** bochecha has joined #buildstream | 09:57 | |
| tristan | tlater, I think we can have our cake and eat it too in some way, for instance if we want a default mode of operation without specifying --directory, we can still have a separate `--tar` option | 09:59 |
| tlater | That seems like it would be confusing. I can imagine myself going "neat, a tar output flag" and accidentally freeze my terminal dumping a 2G tarfile... | 10:00 |
| ssssam[m] | that's the UNIX philosophy though tlater | 10:00 |
| tristan | tlater, a unixy way of expressing it would also be to specify `-` as the output location, which is usually taken to mean stdout or stdin, depending | 10:00 |
| tiago | SIGINT to the rescue | 10:00 |
| *** jennis has joined #buildstream | 10:00 | |
| tlater | Ah, I forgot about '-' | 10:01 |
| tlater | Yeah, that's neater :) | 10:01 |
| tristan | I'm not sure most people will find that intuitive, even if I do | 10:01 |
| tlater | I'd just dislike the default to be dumping to stdout - this always annoys me in other CLI tools | 10:02 |
| tlater | (Although I do know that technically that's the unix way) | 10:03 |
| ssssam[m] | you have to opt in to typing --tar, though | 10:04 |
| ssssam[m] | if `bst checkout` wrote binary data to stdout by default, i'd find it annoying | 10:04 |
| ssssam[m] | but to discover the --tar option you have to have read some kind of manual, and the manual should clearly state "write tar data to stdout" | 10:05 |
| tlater | Yep, and now I find it annoying that `bst checkout --tar` does something different than `bst checkout` | 10:05 |
| tlater | But if `bst checkout -` and `bst checkout --tar -` dump to stdout that does everything quite intuitively IMO | 10:05 |
| ssssam[m] | what would `bst checkout -` do ? | 10:06 |
| tlater | Dump the files in the checkout to stdout - probably not particularly useful, but at least we stay consistent. | 10:06 |
| gitlab-br-bot | buildstream: issue #475 ("Track local files and patches") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/475 | 10:06 |
| tiago | Reiterating my point, I see checking out an artefact and archiving it as two distinct actions, and thus they should be two different subcommands. The '-' is used in replacement of passing filenames, but what checkout does is to create a directory tree | 10:06 |
| tlater | Hm, I suppose there's no real way to dump a directory | 10:07 |
| ssssam[m] | the standard way is to create a tarfile, indeed | 10:07 |
| tristan | tiago, sorry i also skimmed over that, but I really disagree; checking out extracts the data of an artifact; whether you do that in a directory or a tarfile or tar stream I think is an option | 10:12 |
| tristan | the data you are checking out is the same, and is technically more correct using the proposed --tar option | 10:12 |
| tristan | using the --directory option will always be fragile in comparison | 10:13 |
| tristan | ssssam[m], tlater after looking at these options, my personal vote would be a `--tar` option with a filename argument, and specifying a `-` filename for `--tar` puts it in stdout | 10:13 |
| tristan | from a UX perspective | 10:14 |
| tristan | To illustrate why --directory is fragile: Imagine you are checking out a linux system build as a regular user on a windows machine/filesystem... it wont be possible to give you all the ownsership bits and extended fs attributes etc with `--directory` | 10:16 |
| rdale | what forces the bare git repos under buildstream/sources/git to be updated, so the when a build element corresponding to the repo is built, the up to date branch will be fetched - is that what the '--track-all' build option does? | 10:33 |
| juergbi | rdale: yes, bst track (tracking without building) or bst build --track* is used for this | 10:35 |
| rdale | ok thanks | 10:35 |
| gitlab-br-bot | buildstream: merge request (328-support-for-downloading-sources-from-mirrors->master: Resolve "Support for downloading sources from mirrors") #404 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/404 | 10:43 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 10:59 |
| tlater | tristan: Ok, I think the branch is back in a working state | 11:03 |
| tlater | There are two failing tests, but I've deferred fixing those until post-juergbi's branch | 11:03 |
| tlater | His branch will make that easier/obsolete :) | 11:03 |
| gitlab-br-bot | buildstream: merge request (valentindavid/331_include->master: WIP: Add support for include in project.conf) #471 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/471 | 11:08 |
| jonathanmaw | tristan: I've read your "unblocking feature branches" E-mail and am trying to figure out how it applies to my feature branch https://gitlab.com/BuildStream/buildstream/merge_requests/404 . Do I understand correctly that the new rules for what needs review come into force once the bst-1.2 branch has been made? | 11:17 |
| tristan | jonathanmaw, subtle difference here; while the new rules do apply immediately, master is not open for feature development except for a few features (the 4 features we discussed in the BoF) | 11:27 |
| tristan | jonathanmaw, master will be open for 1.3.x material by wednesday, whether we are able to land those features by then or not, though | 11:27 |
| tristan | tlater, ok great ! lets at least try this against freedesktop-sdk | 11:29 |
| tlater | Has the workspace format changed again since I started, or why would you need a separate branch for this? | 11:30 |
| tristan | eh ? | 11:30 |
| tristan | workspace ? I dont see how this is related | 11:30 |
| tlater | You mentioned you wanted to branch a branch for the test | 11:30 |
| tristan | Right, there are 2 branches for the test | 11:30 |
| tlater | I think the only versioned format is workspaces? | 11:30 |
| tristan | A.) A branch based on your branch which bumps BST_CORE_ARTIFACT_VERSION... ensuring that we trigger a full rebuild for the test | 11:31 |
| tristan | B.) A branch of freedesktop-sdk which simply points the buildstream version to the branch created, so that a build runs on gitlab | 11:32 |
| tristan | tlater, just basically a hack to test it, you have another idea ? | 11:32 |
| tlater | Ah, no, nevermind | 11:32 |
| tristan | I mean, of course running it locally helps too | 11:32 |
| tlater | I didn't realize we'd have to change artifact versions | 11:32 |
| tristan | well, it's just a trick :) | 11:32 |
| tlater | s/have to/want to/ | 11:33 |
| tristan | tlater, your branch doesnt cause a need for it, of course | 11:33 |
| tristan | so we wouldnt merge that :) | 11:33 |
| tlater | Yep, I get it now :) | 11:33 |
| jonathanmaw | tristan: ok, so I should wait until wednesday to try and get my feature branch merged | 11:36 |
| jonathanmaw | though since my other branch, https://gitlab.com/BuildStream/buildstream/merge_requests/528, is bugfixing, not features, I can still merge that | 11:37 |
| tristan | jonathanmaw, that's exactly right :) | 11:38 |
| tristan | We might pull in some other features if they are not dangerous, e.g. it would be nice to get Ed's new `remote` plugin into 1.2 | 11:39 |
| *** solid_black has joined #buildstream | 11:41 | |
| *** solid_black has left #buildstream | 11:43 | |
| alatiera | Is it possible to set the bst cache to another path? My ssd is running out of space and I got a 1tb hdd sitting there | 11:44 |
| albfan[m] | alatiera: XDG_CACHE_HOME. I do for same use case as you | 11:46 |
| alatiera | albfan: I guess I would have to alias bst to XDG_CACHE_HOME="foo/bar" bst then :P | 11:47 |
| alatiera | though I was hoping for a cofig key | 11:48 |
| tristan | alatiera, that works, but I would recommend setting both of the `artifactdir` and `builddir` in a ~/.config/buildstream.conf file | 11:48 |
| albfan[m] | alatiera: I launch that way to test gnome-build-meta | 11:48 |
| tristan | alatiera, if you have downloaded a lot of git repos and tarballs, you would keep them that way | 11:48 |
| tristan | only as a matter of reusing the source cache directory | 11:49 |
| alatiera | tristan: exactly what I was looking for, thanks a lot! | 11:49 |
| alatiera | albfan: I am playing around with gnome-build-meta too atm | 11:50 |
| tristan | alatiera, this may help: http://buildstream.gitlab.io/buildstream/using_config.html#configuration-file | 11:50 |
| alatiera | albfan: trying to see if we could purge all the cross-distro CI's and just have a reproducible bst job | 11:50 |
| tristan | alatiera, note that it is actually important to have the builddir and artifactdir on the same partition (we should probably only have a single configuration for those) | 11:50 |
| tristan | because otherwise hardlinks will fallback to copies, which is a real pain | 11:51 |
| alatiera | tristan: acknowledged, thanks! | 11:52 |
| albfan[m] | alatiera: nice! | 11:52 |
| tristan | tlater, ok so this is running now... https://gitlab.com/freedesktop-sdk/freedesktop-sdk/pipelines/25840076 | 12:00 |
| tristan | tlater, I've created the "testing/local-cache-expiry" branch in both repos, and the freedesktop-sdk one refers to the symbolic branch instead of a commit sha | 12:00 |
| tristan | tlater, So it should be possible to like, just rebase BuildStream's "testing/local-cache-expiry" against your branch and manually trigger a pipeline on freedesktop-sdk to re-trigger the test | 12:01 |
| *** leopi has quit IRC | 12:05 | |
| tlater | tristan: Thanks! | 12:14 |
| tristan | weird https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/82002380 | 12:15 |
| tristan | jjardon, are you aware of that issue ? looks like freedesktop-sdk builds fail while GPG verifying an ostree download | 12:16 |
| tristan | with... too many open files | 12:16 |
| tlater | tristan: If not, perhaps we're spawning more jobs than we should do, or something of that sort | 12:18 |
| tlater | Seems very weird, though | 12:18 |
| tlater | Hmm, all other fetches seem to run fine though. | 12:20 |
| tristan | Looks like it failed for both x86 builds... I re-triggered x86_64 | 12:22 |
| tristan | tlater, also looking at the arm build: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/82002386, I'm seeing a lot of cache cleanup jobs... I think | 12:23 |
| tristan | e.g. [00:00:00][][] SUCCESS cache_size.280.log | 12:24 |
| tristan | tlater, they seem to be happening in series, like after every pull, even if the pull did not pull anything | 12:24 |
| tlater | tristan: Not cleanup jobs | 12:24 |
| tristan | Cache usage calculation jobs ? | 12:25 |
| tlater | Those jobs trigger if we suspect that we have too much data in the cache | 12:25 |
| tlater | Since pull jobs have no way of reporting how much data they may have added, they will always trigger one currently - perhaps that's something to optimize though | 12:25 |
| tlater | Yeah, it's a lot of them... | 12:25 |
| tristan | Seems to be a lot of them, anyway I think we can improve this after | 12:26 |
| tristan | I dont like seeing that many of them in the log though hehe | 12:26 |
| tlater | Yep, this will definitely need to be optimized | 12:26 |
| tristan | signal noise ratio not great | 12:26 |
| tlater | I should just disable log messages for these jobs in general. | 12:26 |
| tlater | Perhaps keep them only for actual cleanup jobs | 12:27 |
| *** leopi has joined #buildstream | 12:28 | |
| tristan | tlater, it sounds reasonable to not have the start/success messages here, anyway a failure/bug should still appear in the log | 12:28 |
| tlater | Oh, yeah :S | 12:29 |
| tristan | tlater, but I dont want to block on that | 12:29 |
| tristan | tlater, better that you also build freedesktop-sdk with the branch | 12:29 |
| tristan | and I'll also try launching a build overnight here | 12:29 |
| tristan | see what happens | 12:29 |
| tristan | I'll do it with gnome-build-meta and build epiphany or smth | 12:30 |
| tlater | Eugh, this poor laptop really needs more ram | 12:34 |
| tristan | I usually manage to squeeze a build into about 7GB :) | 12:39 |
| tristan | peaking around webkit | 12:39 |
| tristan | tlater, I think this means that it passed with a retry: https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/82010341 ... since it's now building, it must have downloaded the SDK it uses | 12:44 |
| tristan | I think it's quite unrelated to your branch anyway | 12:44 |
| tlater | Hmm | 12:44 |
| tlater | That's scary | 12:44 |
| tristan | tlater, well, I think it demonstrates that GPGME could be more fault tolerant ? | 12:45 |
| tlater | Hmm, I suppose if we were launching too many jobs we'd see it in some other form as well | 12:46 |
| tristan | Builds can fail for lack of RAM or over parallelization as well yeah | 12:46 |
| tristan | (over parallelization usually lending to requiring more RAM) | 12:47 |
| tlater | I was moreso thinking along the lines of scheduling the same job many times over :) | 12:47 |
| tristan | but I'm pretty stumped about too many files, there should be the same amount of files in the SDK both times you try to verify it | 12:47 |
| tristan | and the limit on open files as I understand is a per-process limit | 12:47 |
| tlater | Oh, alright, that is even stranger then | 12:47 |
| tlater | But does seem unrelated to my changes. | 12:48 |
| tlater | Perhaps I'm paranoid, but I'll steel keep an eye on this in local builds. | 12:48 |
| *** tristan has quit IRC | 12:53 | |
| *** tristan has joined #buildstream | 13:01 | |
| tristan | tlater, a start would be to not trigger the cache size check thing in the case that a Pull job completes with the skip condition | 13:08 |
| tristan | tlater, i.e., a pull job that did not pull anything after all | 13:08 |
| tlater | Yep, that's definitely something that needs to happen | 13:08 |
| tristan | I'm pretty sure I'm seeing these jobs happen regardless | 13:09 |
| tristan | after that's done it will be reduced quite a bit | 13:09 |
| tlater | Once I get my history in shape I'll get to fixing that :) | 13:09 |
| tlater | I really need to get better at this history thing... | 13:09 |
| tristan | tlater, this is a pretty huge branch so... in that case I usually like to float preparational changes which are good for master up to the beginning of the branch, apply them separately and consequently remove some noise from the branch | 13:11 |
| tlater | Hm, I'm already doing that. The problem is that I've essentially reworked the feature again, which means the changes are intertwined between commits. | 13:11 |
| tristan | tlater, also, lets focus on having every commit pass it's tests rather than making everything nice and separate; if we have a couple of mega commits in this one, it's not a huge deal I think | 13:11 |
| tristan | sometimes I setup a separate branch and use `git reset` and just restage things in that case | 13:12 |
| tlater | That might well be a good option here. | 13:12 |
| tristan | separate branch is key when doing git reset :) | 13:13 |
| tlater | Yeah, I can see it being hard to figure out what happened with no history at all | 13:14 |
| *** leopi has quit IRC | 13:32 | |
| *** jcampbell has joined #buildstream | 13:39 | |
| gitlab-br-bot | buildstream: merge request (valentindavid/331_include->master: WIP: Add support for include in project.conf) #471 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/471 | 13:44 |
| *** noisecell has quit IRC | 13:55 | |
| gitlab-br-bot | buildstream: merge request (138-aborting-bst-push-command-causes-stack-trace-3->master: WIP: Resolve "aborting bst push command causes stack trace") #514 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/514 | 14:03 |
| *** noisecell has joined #buildstream | 14:04 | |
| *** leopi has joined #buildstream | 14:05 | |
| gitlab-br-bot | buildstream: merge request (juerg/googlecas->master: WIP: Remote Execution CAS-based artifact cache) #337 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/337 | 14:33 |
| tpollard | is anyone using buildstream natively on an ubuntu host? | 14:53 |
| finn | jmac ^^ | 14:53 |
| jmac | I am | 14:53 |
| tpollard | I'm currently re-purposing an old build machine & was going to put 18.04 on it | 14:54 |
| tpollard | & then noticed it's not under the install from source section | 14:54 |
| jmac | I believe the instructions are identical to Debian 9 | 14:55 |
| tpollard | I suspected so, ta jmac | 14:55 |
| jmac | I don't remember having to do anything special, but if you find any problems, please report them | 14:55 |
| jmac | Also if you get it running we should add a note that it's the same as Debian | 14:55 |
| noisecell | tpollard, there is also a merge request with the ubuntu information: https://gitlab.com/BuildStream/buildstream/merge_requests/525/diffs | 14:57 |
| jmac | Nice one noisecell & Phil | 14:58 |
| jmac | Although Ubuntu 17.04 is already EOL so that needs updating | 14:59 |
| noisecell | https://gitlab.com/BuildStream/buildstream/merge_requests/525/diffs?commit_id=e74f891cbfb7ec1e971f56499a6a2f41a9aeb57f -- I think this are the instructions for >=17.10 so 18.04 should be the same | 14:59 |
| noisecell | tpollard, ^^ | 14:59 |
| jmac | Yeah, it does say "17.04 or higher" but referencing 17.04 seems dated somehow | 15:00 |
| *** finn has quit IRC | 15:02 | |
| gitlab-br-bot | buildstream: merge request (425-add-a-deps-flag-to-bst-checkout->master: WIP: Resolve "Add a `--deps` flag to `bst checkout`") #544 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/544 | 15:07 |
| gitlab-br-bot | buildstream: merge request (425-add-a-deps-flag-to-bst-checkout->master: WIP: Resolve "Add a `--deps` flag to `bst checkout`") #544 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/544 | 15:09 |
| gitlab-br-bot | buildstream: merge request (add_flag_to_checkout->master: Add a `--deps` flag to `bst checkout`) #545 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/545 | 15:13 |
| Nexus | Please could someone review !545 for me please? | 15:14 |
| * skullman is having a read | 15:15 | |
| gitlab-br-bot | buildstream: merge request (add_flag_to_checkout->master: Add a `--deps` flag to `bst checkout`) #545 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/545 | 15:17 |
| *** leopi has quit IRC | 15:22 | |
| jmac | Looks good in that it'll fix #425 but I'm not clear on why you'd want an element without its runtime dependencies | 15:23 |
| * skullman recalls from discussion elsewhere that it was for applications which there were already guarantees that the element's files would be binary compatible with where it goes after it has been checked out | 15:25 | |
| skullman | not sure if I've explained that sufficiently | 15:26 |
| jmac | Yes, that makes sense | 15:26 |
| tlater | I wonder if this should be tristate | 15:27 |
| tlater | No dependencies, runtime dependencies, all dependencies | 15:27 |
| gitlab-br-bot | buildstream: merge request (add_flag_to_checkout->master: Add a `--deps` flag to `bst checkout`) #545 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/545 | 15:29 |
| gitlab-br-bot | buildstream: merge request (add_flag_to_checkout->master: Add a `--deps` flag to `bst checkout`) #545 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/545 | 15:29 |
| Nexus | general thoughts on an if-else assert vs assert os.path.exists(filename) == (foo == "bar") | 15:31 |
| Nexus | ? | 15:31 |
| jmac | I find the if-else version clearer | 15:31 |
| skullman | has the benefit of the line number letting you know which bit failed without the more detailed assert deconstruction | 15:32 |
| *** cs_shadow has joined #buildstream | 15:37 | |
| *** Kinnison has joined #buildstream | 15:38 | |
| Nexus | ok, i'll leave that comment open for now until i can get some feedback off'f either juergbi or tristan | 15:38 |
| Nexus | otherwise, should be ready for merge | 15:39 |
| Nexus | unless tlater wants to REEEEALLY push for his tristate | 15:39 |
| Nexus | but i think he just likes things having trista in them | 15:39 |
| tlater | Heh | 15:39 |
| tlater | Give me a minute to ponder why I wanted this feature again... | 15:40 |
| juergbi | I think the if-else is slightly easier to understand but I'm fine with either, no strong opinion | 15:41 |
| *** bochecha has quit IRC | 15:41 | |
| tlater | Yeah, I don't think we need a tristate either | 15:42 |
| tlater | I don't really see a use case | 15:43 |
| juergbi | (my comment was just about the assert, not about tristate) | 15:43 |
| juergbi | I can't think of a use case for including build-only dependencies in the checkout | 15:44 |
| juergbi | however, maybe we should actually have 'run' and 'none' instead of 'all' and 'none' for consistency | 15:45 |
| juergbi | (consistency with bst show) | 15:45 |
| tlater | +1 | 15:46 |
| gitlab-br-bot | buildstream: merge request (jonathan/fix-filter->master: Jonathan/fix filter) #528 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/528 | 15:49 |
| *** leopi has joined #buildstream | 15:53 | |
| jonathanmaw | tristan: My bugfix branch for filter elements has extended as far as changes to the command-line interface, since it was that or leave some unpleasant surprises for anyone tracking through the code | 15:53 |
| jonathanmaw | i.e. `bst track`'s '--deps' arg uses 'redirect' instead of 'none' (with the default changed, and 'none' being changed to 'redirect' to maintain backward compatibility) | 15:54 |
| jonathanmaw | Is that still acceptable as a bugfix, or should I make sure that the public-facing API is unchanged? | 15:55 |
| juergbi | jonathanmaw: oh, as per my first comment, I tend to be against exposing this in the CLI | 15:57 |
| juergbi | I thought the latest rename was just about the internal enum | 15:57 |
| jonathanmaw | juergbi: ah, okay | 15:58 |
| juergbi | sorry for the confusino | 15:58 |
| juergbi | *confusion | 15:58 |
| jonathanmaw | I'll do the sneaky substitution of "none" to "redirect" | 15:58 |
| tiago | Is it possible to run only a single test in the test suite? | 15:59 |
| jonathanmaw | and raise an issue of whether that should be changed | 15:59 |
| juergbi | tiago: yes, e.g., ./setup.py test --addopts tests/frontend/push.py::test_artifact_too_large | 15:59 |
| tiago | aahh, thanks | 15:59 |
| *** finn has joined #buildstream | 15:59 | |
| juergbi | you can also use -k to specify tests by string keyword expressions | 16:01 |
| juergbi | https://docs.pytest.org/en/latest/usage.html | 16:01 |
| tlater | tiago: Unfortunately the -k expression currently also triggers pylint | 16:02 |
| tlater | And running explicitly only a single test *won't* run pylint | 16:02 |
| tlater | Which is a bit annoying... | 16:02 |
| gitlab-br-bot | buildstream: merge request (jonathan/fix-filter->master: Jonathan/fix filter) #528 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/528 | 16:12 |
| gitlab-br-bot | buildstream: merge request (juerg/googlecas->master: WIP: Remote Execution CAS-based artifact cache) #337 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/337 | 16:25 |
| tiago | Is there any test that captures the stdout | 16:31 |
| tiago | ah, nevermind | 16:33 |
| *** finn_ has joined #buildstream | 16:34 | |
| *** finn has quit IRC | 16:34 | |
| gitlab-br-bot | buildstream: merge request (jonathan/fix-filter->master: Jonathan/fix filter) #528 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/528 | 16:45 |
| *** bethw has quit IRC | 16:50 | |
| *** coldtom has quit IRC | 16:53 | |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 16:56 |
| tiago | The Cli class used for tests is not coupling very well when the output is binary data | 16:57 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 16:57 |
| *** jonathanmaw has quit IRC | 17:00 | |
| *** finn_ is now known as finn | 17:08 | |
| *** finn has quit IRC | 17:16 | |
| *** finn has joined #buildstream | 17:16 | |
| *** finn has quit IRC | 17:37 | |
| *** finn has joined #buildstream | 17:37 | |
| gitlab-br-bot | buildstream: merge request (valentindavid/331_include->master: WIP: Add support for include in project.conf) #471 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/471 | 17:38 |
| *** leopi has quit IRC | 17:40 | |
| *** leopi has joined #buildstream | 17:43 | |
| *** finn has quit IRC | 17:45 | |
| *** ernestask has quit IRC | 17:59 | |
| *** cs_shadow has quit IRC | 18:03 | |
| *** leopi has quit IRC | 18:23 | |
| gitlab-br-bot | buildstream: merge request (valentindavid/331_include->master: WIP: Add support for include in project.conf) #471 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/471 | 18:27 |
| *** cs_shadow has joined #buildstream | 19:08 | |
| *** tristan has quit IRC | 20:32 | |
| *** Prince781 has joined #buildstream | 23:12 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!