IRC logs for #buildstream for Monday, 2018-04-30

*** Prince781 has quit IRC01:18
*** Prince781 has joined #buildstream02:30
*** Prince781 has quit IRC03:28
*** tristan has quit IRC04:23
*** tristan has joined #buildstream04:28
*** Prince781 has joined #buildstream04:50
*** Prince781 has quit IRC04:54
*** jsgrant has quit IRC04:56
*** Prince781 has joined #buildstream05:25
*** Prince781 has quit IRC05:49
*** toscalix has joined #buildstream07:30
*** bochecha_ has quit IRC07:43
*** bochecha_ has joined #buildstream07:47
*** jennis has joined #buildstream08:04
paulsherwoodplease can we fix the initial text at https://wiki.gnome.org/Projects/BuildStream to match the text at https://gitlab.com/BuildStream/buildstream/blob/master/README.rst ?08:16
* paulsherwood has just had earache from a colleague complaining about the gnome text (which is the first thing google will offer)08:17
tristanpaulsherwood, done08:21
*** bethw has joined #buildstream08:23
toscalixpaulsherwood: I will put effort on the front page/product charter.08:23
paulsherwoodtvm08:24
toscalixFor those of you who do not know me, btw, I will be helping tristan on buildstream. You will see a mail on the mailing list later today about it.08:26
toscalixI work at Codethink as consultant. This is not my first collaboration with GNOME. I am new to Buildstream though so it will take me some time to get into details08:29
*** jonathanmaw has joined #buildstream08:38
paulsherwoodtoscalix: welcome aboard :)08:41
*** solid_black has joined #buildstream08:41
toscalixthanks08:41
tristanwelcome toscalix :D08:42
toscalixI will attend to today's IRC monthly meeting08:43
tlatertoscalix: It's still early enough that time zones probably aren't messing with me here, did the meeting date change?08:50
toscalixthe IRC meeting is tomorrow, sorry08:52
toscalixand now that I think about it, it is bank holiday in many countries08:52
tristanHmmm08:52
tristanShould we try to punt it ? it's a bit late for this08:52
tristanMaybe we should reschedule the holiday instead ?08:52
tlaterYep, that sounds preferable08:53
tlaterMove it to a day around Christmas?08:53
toscalixlet's see what laurence thinks about moving it08:54
tristanI think it would be good to bump it by a week08:55
tristannext tuesday08:55
tristanin fact, the majority of the attendees are in the UK08:56
tristanSo I think it makes sense08:56
*** dominic has joined #buildstream09:00
tlatertristan: Tomorrow is not a bank holiday in the UK :)09:01
tristanI am confused09:01
tristanWhy do people think it's a holiday ?09:01
*** tiago has quit IRC09:01
*** solid_black has quit IRC09:01
tristanYou know, since I dont work at a bank, I really don't know these things in advance09:01
tlaterIt's in a lot of other countries, not the UK, as far as I am aware09:01
*** tiago has joined #buildstream09:02
*** solid_black has joined #buildstream09:02
gitlab-br-botbuildstream: merge request (juerg/push-duplicate->master: Push improvements) #442 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/44209:07
toscalixah, not in UK, true09:08
milloniis the first monday of may a bank holiday in the uk?09:25
milloniif so people might get confused cause this monday is the 30th09:25
toscalixmilloni: it is09:28
toscalixthat was my confusion too.09:28
tlaterHm, usually not following the international norm is weird, but I suppose it's handier to have a long weekend than a random day off09:29
tristanjonathanmaw, juergbi eek.... ummm... was it necessary to change the initialization process for redirection of workspace commands ?09:50
tristanActually, this is in flux, all of that changed code is gone in my branch which aims to sort out this pipeline mess09:50
tristanSo, I expect I'll have to look closely at what that patch does and factor it in09:51
jonathanmawtristan: I think we can avoid doing that if we partially initialize the app once, then fully initialize it again later09:53
jonathanmawbut that sounds like it could have unintended consequences09:53
*** cs_shadow has joined #buildstream09:54
Nexustristan: please could you expand on what you said in your reply to my email about "just address what we want to download separately" I can't quite follow what you mean09:54
tristanNexus, artifacts are named blobs, which we transfer to and fro; they are addressed by their artifact name, which is composed of a project name, element name and cache key09:55
tristanNexus, it is not possible to tell the artifact server "Give me this part of that blob"09:56
tristanInstead, we need to address the blobs separately09:56
tristanWith ostree this means making separate commits for those blobs09:56
tristanAs Sander points out, there will/may be a different way to do that with CAS09:56
tristanBut that should be a separate thing, because CAS should still support separated naming of blobs (as it must support addressing of artifacts anyway)09:57
tristanAnd I dont want to block things because it might be more elegant to do it a different way with CAS09:57
tristanRather, we can change that once we have CAS09:58
tristanjonathanmaw, So just to paint a picture (I have meeting in few minutes...) App.initialize() now just creates the Stream object and some basic things, it no longer create the Pipeline at all, and partially_initialized() is not needed anymore09:59
tristanjonathanmaw, all pipeline manipulation stuff has moved to Stream.build(), Stream.fetch(), Stream.track() etc10:00
tristanSo the Stream() is the main core frontend facing API and does the highlevel core business logic, while Pipeline is only for loading and manipulating lists of elements10:00
laurencetoscalix, I don't think we need to move tomorrow's meeting. vast majority of regular attendees are in the UK. exceptions being juergbi, tristan and yourself, who are all expected to turn up10:17
jonathanmawtristan: ok, do you think your tristan/pipeline-refactor branch is stable enough that I can try and adapt my changes to that branch, or should I wait until it's merged into master?10:18
*** solid_black has quit IRC10:18
jennisIs there any reason / has there ever been a discussion as to why we favour os.path to pathlib.Path?10:30
* jennis stumbled across pathlib.Path this weekend and is curious 10:30
tlaterjennis: You'd have to use pathlib.Path throughout the entire code base to gain any advantage10:31
tlaterI find trying to use it adds a *bit* too much boilerplate still10:31
jennisahh, I guess we're in too deep for that10:32
* skullman tried it out when it was announced and had issues with its unicode handling, but they may have been fixed10:32
tlaterjennis: We do use it occasionally for `.parents` - I don't think there is an os.* equivalent for that.10:33
tristanjonathanmaw, My next push to it will make the biggest change... tests are *almost* all passing10:33
tristanjonathanmaw, Not sure if it answers your question, so I'll share a bit my plan...10:34
jennistlater, oh really? Guess I'm not grepping properly then...10:34
tristanjonathanmaw, basically; I've done all the restructuring so that Pipeline is internal and owned/used by Stream... I still need to do the actual refactoring of Pipeline *itself*10:35
tristanjonathanmaw, and besides this, I'm also going to hide the scheduler from the frontend behind Stream(), such that the frontend really only sees Context, Project and Stream10:35
tristanI've also discovered that, miraculously enough, `bst source-bundle` seems to "still work"10:39
* tlater clearly wrote very high quality code there10:40
tristanjennis, tlater I agree it adds boiler plate, although I think we use pathlib for heavy lifting, for implementation of utils.glob() (or we did use it initially, maybe it changed for a straight regex)10:40
tristanBut... one thing to take into account10:40
tristanjennis, tlater ... and I suspect this might bite us one day... if we get to supporting many platforms, we get it wrong in quite a few places10:41
tlaterYep... Part of preparing bst to run on, say, Windows, would probably be to start using pathlib...10:42
tristanI.e., we use os.path.join() for loading anything on the host, that is correct... *however*, we should not be making assumptions that host `os.path.join()` is correct for constructing strings which might be used in the sandbox10:42
jennistristan, yes the impression I got was that pathlib.Path is much more favourable to use as it works well cross-platform10:42
tristanos.path.join() should be fine on windows10:42
tristanbut maybe not everything, not sure...10:42
tlatertristan: I think we have other, bigger problems for windows adoption, though10:43
tristananyway, windows is a long shot... although it would be really fun to get the source code for the MSVC compiler and base C runtime and kernel there, and bootstrap the next big version of windows with BuildStream :D10:43
tlatertristan: On that note, docker would make for an interesting sandbox implementation10:44
tristanthe real utility of running natively on windows I think is still for building linux embedded on windows machines10:45
tristanAnd I think the WSL or whatever it's called thing that is available since windows 10 will mostly allow that10:46
tristanso we could have a sandbox for that, not sure if docker is the right way to do that, maybe it is10:46
tlaterHm, interesting thought, I wonder if the "WSL" has some form of an API?10:47
* tlater doubts it, but *maybe*10:47
tristanyeah it should, juergbi looked into that a while back I'm pretty sure it does10:47
tristantlater, we only hear about "It supports Ubuntu" or something high level market speak10:48
tristanAnd probably it's hidden underneath a crap ton of user friendly point and click annoyances10:48
tlaterI assumed it would mostly be end-user facing, so I figure it might be hard to use for any real programming10:49
tristanbut I would totally expect that somewhere beneath that whole mess, is an actual API :)10:49
tristanI would want to use it directly, and stage only runtimes we import and build with BuildStream10:49
tristanand not have to "Run BuildStream in Ubuntu in Windows" or something redundant like that10:49
tristantlater, it's mostly just a question about how limited the implemented syscall interface is, iirc, but it must be fairly well implemented for it to run ubuntu10:50
tristanmaybe we would not have user namespaces though :)10:50
* tlater is really curious about this :)10:51
tlaterThough docker might still be a better idea, given that it would allow mac support as well.10:51
tristantlater, an interesting breakage I noticed while looking at the pipeline refactor: `bst fetch --track --except` is totally broken :)10:57
tristanit probably never worked as advertised, actually10:58
tlaterErk10:58
gitlab-br-botbuildstream: merge request (juerg/push-duplicate->master: Push improvements) #442 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/44210:58
gitlab-br-botbuildstream: issue #325 ("bst is pushing artifact it just have pulled") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/32511:08
gitlab-br-botbuildstream: merge request (juerg/push-duplicate->master: Push improvements) #442 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/44211:08
tristanjonathanmaw, I've pushed the bigger change now, it's being tested here: https://gitlab.com/BuildStream/buildstream/pipelines/2129000011:16
tristanjonathanmaw, you might look and see11:16
tristanAs I said, I still have to untangle the Pipeline(), it's still using the same weird initialization paths, but now the damage is localized11:17
tristanAlso I still mean to change a bunch of subtle stuff, like hiding the Scheduler away and making that a private detail of Stream11:17
tristanI also still didnt remove the now pointless App.partially_initialized()11:18
jjardonhttps://gitlab.com/BuildStream/buildstream/issues/325 \o/ thanks a lot juergbi !11:20
tristanjonathanmaw, On a separate topic, ... valentind is working on the `bst mirror` stuff while as I understand you will be getting back to the client side of source mirroring, you will probably both want to sync up at some point11:23
jonathanmawtristan: yep11:23
tristanI've suggested using the mailing list for now, as we're at the beginning and probably ironing out kinks in the architecture11:24
valentindjonathanmaw. Yes. Paths for mirror generated by "bst mirror" should be generated by plugin. And we need a support of incremental namespacing.11:24
* tristan now understands what valentind was talking about with the word "namespaces" :)11:25
valentindSorry tristan, I thought you used that term.11:26
tristanvalentind, Oh maybe :) no worry !11:40
tristanvalentind, I just remember being confused last week, cause namespaces can mean a lot of things :)11:41
valentindtristan, yes.11:41
valentindjonathanmaw, I sent an email on the mailing list about that.11:57
jennisHey guys, has anyone got a chance to review !421 expiring the remote share? I think it's only 1 commit away from ready. Regarding the last commit, it's WIP I'd be grateful for some help?12:57
jennis*and I'd12:57
tristanjennis, I was tied up last couple days deep in refactor... and haven't looked yet because I'm hoping for juergbi to land the scheduling changes ahead of that (as you probably noticed in issue comments)13:02
jennistristan, yes I did. No worries. I'd just like to get this "artifact too large" case dealt with as it's infuriating me13:05
jennisAny idea when those scheduling changes are likely to land? juergbi?13:06
juergbijennis: hopefully really soon13:15
jennisnice13:17
juergbitristan: in case you haven't realized, you're accessing private pipeline API in _stream.py13:41
NexusI need to understand more about remote artifact caches, who's the best person for me to talk to13:42
tristanjuergbi, there are some loose ends13:43
tristanjuergbi, it's not ready for review :D13:43
juergbiyes, I figured. just thought I mention it13:43
tristanjuergbi, I am also doing something disgusting from app.py, I needed to tape a couple things in place whilst moving the underlying machinery :)13:43
tristan_pipeline.py is going to change almost completely, and a nice API will emerge, but a ton of things needed to happen for that to even be possible13:44
juergbiright. it's a bit of a conflict area with my changes, let's hope it won't be too bad13:45
tristanIts the high level initialization, which changes almost completely; the frontend will have a very limited view of the core13:46
juergbiNexus: just ask in here and I or someone else will try to answer13:46
tristanAll of those arguments to app.initialized() have disappeared13:46
tristanand finally I removed fetch_subprojects from Context too13:46
Nexusjuergbi: generally, i know nothing about that similatities or differencesm so i need to ask everything13:47
tristanjuergbi, which changes in particular ? to be honest; while I doubt a magic git rebase merge would have any chance of success - I am confident that applying whatever changes you have to the new core will be straight forward13:48
juergbitristan: the more dynamic handling of build dependencies13:48
tristanAh, that requires frontend initialization specifications ?13:48
tristanI dont think it would much13:48
juergbino, it's more core pipeline stuff13:49
juergbiso it might not be too bad13:49
tristanAh, yeah; some things there will change; the Element stuff remains untouched, or almost completely untouched13:49
tristanI doubt it will be bad13:49
tristanWhatever changes you made to _pipeline.py will have to be re-applied manually13:50
jennisNexus, the remote cache is almost identical to ~/.cache/buildstream/artifacts/ostree/13:50
tristanunfortunately that was the last part, though; so you will have to wait until tomorrow to see the brave new world :)13:50
juergbihehe13:51
jennisOn the server, this will be within /data/artifacts13:51
tristanI am essentially thinking, the pipeline will do loading in one stage, without knowledge of what the target lists are for... and Stream will feed those targets back into functions which do exotic sorting and filterings, and those results will get fed into the Queues and the scheduler13:52
tristanAnd the artifact cache initializations will be removed from Pipeline as well13:52
jennisWhen you set up a server, we instruct whoever is doing this to create a user, aptly named "artifacts" who has write permissions for this directory13:52
* tristan misstepped and did that prematurely, then realized that artifact cache initialization needs to happen *in between*13:52
tristanprobably, your branch will end up removing a lot of that13:53
tristanalthough loading the configs still needs to be done, after recursive loading, such that the recursive project configurations are already present13:53
jennisOne thing I found the other day was that, although these cache's look identical, the disk space taken up by the local cache is more. tristan informed me that this was due to sending compressed objects to the server13:54
tristanjennis, due to using archive-z2 repo mode on the server, yes13:55
jennisNexus ^^ which we do upon initialising the remote cache: `ostree init --mode archive-z2 --repo artifacts`13:56
jennisAlso note that there is no tarcache on the server13:58
*** tristan has quit IRC13:58
laurenceI've sent a mail to the list to update on the role of toscalix14:01
toscalixthank you laurence I will answer it with info about where I plan to help14:02
*** tristan has joined #buildstream14:03
laurencegreat :)14:03
jjardonAre the different "format-version" documented somewhere?14:37
toscalixlaurence: done14:38
toscalixI hope to get feedback14:38
juergbijjardon: there are 'since format version' notes in the documentation for the corresponding config keys14:39
jjardonjuergbi: thanks; is that in the docs somewhere? seems it's only mentioned here as the default paramter: https://buildstream.gitlab.io/buildstream/projectconf.html#builtin-defaults14:43
jjardonbtw, does current buildstream defaults to "format-version: 0" ?14:43
juergbijjardon: an example is at the end of https://buildstream.gitlab.io/buildstream/projectconf.html#ref-storage14:43
jjardonah! here it is: https://buildstream.gitlab.io/buildstream/projectconf.html#format-version14:44
juergbiyes, default is version 0 as per your previous link with the defaults14:44
*** persia has quit IRC15:03
jjardonjuergbi: is ok if I open an issue to document the differences between versions, so it's easy to see why I should use a different format-version? (I think the latest is 9)15:06
*** oknf[m] has joined #buildstream15:06
jjardonlaurence: adds68 valentind https://cache.sdk.freedesktop.org/releases/refs/heads/runtime/org.freedesktop.Sdk/arm/15:15
jjardontristan: ^ (thanks for that sled-8)15:16
valentindjjardon, great!15:16
valentindLet me try it.15:16
adds68woohoo \o/15:16
jjardonoops, wrong channel, sorry still related tough :)15:16
tristanjjardon, I was thinking such a doc could be useful, but should link directly to the features for each introduced "thing", note there will be some gaps (versions which were bumped because of a change in functionality of a feature during unstable cycle)15:17
jjardonvalentind: cheers, please open an issue if you see something wrong15:18
tristanjjardon, currently though, the reverse is completely documented; i.e. every YAML configuration which is not available in version 0, states it's version15:18
tristanif it is not, that is a bug, but I am pretty sure that anything which introduced a version bump, states the version15:19
*** bethw has quit IRC15:20
jjardontristan: mmm, I tihnk state the version in a buildstream project is not the same as the version itself being documented?15:20
tristanjjardon, it's more important I think, i.e. think of it this way: You look at the reference to find how to use features, the reference tells you if a documented feature is only available in a specific version15:21
tristanjjardon, however as I said, such a doc which does the reverse could also be useful15:21
tristani.e. an ordered list which links out to the specific features introduced with each version15:21
tristanwhich (as I also mentioned above), may have gaps15:21
jjardonyup, but still would be better than nothing I think; let me open an issue so we do not forget.15:22
gitlab-br-botbuildstream: issue #381 ("Introduce a new source plugin - `SourceTransform`") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/38115:22
tristanjjardon, the fact there is a version itself, is actually one of the *first* things documented: http://buildstream.gitlab.io/buildstream/projectconf.html#format-version15:23
tristanjjardon, and every feature which requires a version links to that link, with "Available since format version X" text15:23
*** bochecha_ has quit IRC15:23
jjardontristan: tristan that doesnt give me any info why a bst project is using version 3 , 4 or 9. Or why I should upgrade to a new version with new features that can be useful to my project15:24
tristanjjardon, Why would you want to know that, though ?15:24
tristanjjardon, as I said, it *can* be interesting, I'm not against having a page which has an incremental list documenting that, and have thought it could be useful in some way15:25
tristanI.e. if a bst project requires version 3, it is not really up to you to wonder why: it's up to you to upgrade bst and get a bst which supports it15:26
tristanjjardon, if you are on the other hand, adding a feature to an existing project: It's up to you to read the reference about that feature and also update the format-version15:26
tristanfor that part, we have it covered15:26
juergbitristan: maybe an overview of which BuildStream version supports which format version would be useful, though15:28
juergbito know the bst requirement for a project15:28
tristanjuergbi, yes, as I've stated a couple of times above, I'm certainly not against it :)15:29
tristanIt should ideally indicate at least the major point stable release it was introduced in as well15:29
jjardonI see. Still a after quick "$ git grep "since :ref:""  only versions 1,4,6,8 are mentioned at all ; no features have been added in any other versions?15:30
*** Prince781 has joined #buildstream15:30
tristanfor each version... maybe a treeish list starting with 1.2 (because 1.0 iirc came out with version 0).. and all the versions introduced in 1.2... then in next cycle 1.3, we start adding to the list of 1.4 features15:31
tristansomething like that15:31
tristanjjardon, I mentioned above there is a reason for there being "gaps"15:31
tristan "note there will be some gaps (versions which were bumped because of a change in functionality of a feature during unstable cycle)"15:31
tristanjjardon, one of the reasons for that is for instance, we now require version 8 for "ref-storage" (the project.refs feature)15:32
jjardontristan: yeah, Im asking now if that is actually the case for all the versions not mentioned there. Or we simply don't know15:32
tristanjjardon, it was introduced in a different version, but now we store the junction.refs separately, so we use version 8 to indicate that feature15:32
tristanit happened in a few cases15:32
tristanI cant say with 1000% certainty but I've taken very good care to document those15:33
gitlab-br-botbuildstream: issue #382 ("Document changes in format-version in a central place") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/38215:36
tristanNicely named issue :)15:37
*** Prince781_ has joined #buildstream15:38
tristanlets link to it from http://buildstream.gitlab.io/buildstream/projectconf.html#format-version, but put it somewhere at the end of the reference ToC15:38
*** Prince781 has quit IRC15:39
*** Prince781_ is now known as Prince78115:39
* tlater is successfully calculating cache sizes on a separate thread/process/asyncio-thing :)16:17
skullmanneat16:18
tlaterskullman: I think the interface is pretty close to what it will look like... I should push and link you to my MR now, sorry, I had a few meetings in-between and forgot.16:19
* tlater should have something pop up his todo list every hour or so16:20
gitlab-br-botbuildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/34716:25
tlaterskullman: That's the one :) https://gitlab.com/BuildStream/buildstream/merge_requests/34716:26
skullmanta16:26
gitlab-br-botbuildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/34716:28
*** Prince781 has quit IRC16:48
*** toscalix has quit IRC17:00
*** dominic has quit IRC17:12
*** jonathanmaw has quit IRC17:32
*** jonathanmaw has joined #buildstream18:00
*** slaf has joined #buildstream18:55
*** tristan has quit IRC20:12
*** Prince781 has joined #buildstream23:27
*** jsgrant has joined #buildstream23:42
*** Prince781 has quit IRC23:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!