IRC logs for #buildstream for Tuesday, 2018-09-04

*** leopi has joined #buildstream04:33
*** tristan has joined #buildstream06:37
*** alatiera_ has joined #buildstream06:55
*** rdale has joined #buildstream07:38
*** tristan has quit IRC07:49
*** toscalix has joined #buildstream07:53
*** tristan has joined #buildstream08:11
*** jonathanmaw has joined #buildstream08:26
*** solid_black_ has joined #buildstream08:29
*** solid_black__ has joined #buildstream08:31
*** solid_black_ has quit IRC08:33
*** leopi has quit IRC08:40
*** leopi has joined #buildstream08:42
*** tristan has quit IRC08:43
qinustyWhat route are we taking on analytics toscalix, valentind?08:48
toscalixnone for now08:48
toscalixin the coming weeks I will explore08:48
toscalixanallytics on the website08:49
toscalixtelemetry related with the software production comes next. Finally it would be about telemetry on the software itself, in accordance with open source and privacy standards08:50
toscalixI do not think the third step will happen soon08:50
toscalixwe have a long way to go before it comes neccessary to understand how the software is being used in order to improve it. There are several Open Source projects that are doing a good job in this regard. We can evaluate them08:51
toscalixI had a nice conversation about this point in a dinner with people from canonical, debian and opensuse. Each one of them has a different approach08:51
toscalixit will be an interesting discussion to have in the future08:52
toscalixKDE for instance will be discusing about it this year.08:53
toscalixThere was more resistance a few years ago than now since there are better tools now to respect data privacy while collecting data to improve08:54
*** tristan has joined #buildstream08:56
*** ChanServ sets mode: +o tristan09:03
*** bochecha has joined #buildstream09:33
*** dtf has joined #buildstream09:35
*** lachlan has joined #buildstream09:36
qinustytristan, what's happening with the development snapshot badge? Will it work prior to 1.3.1?09:50
gitlab-br-botbuildstream: merge request (jmac/remote_execution_client->master: WIP: Remote execution client) #626 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/62609:59
*** lachlan has quit IRC10:08
*** jonathanmaw_ has joined #buildstream10:17
*** jonathanmaw has quit IRC10:18
WSalmontristan, thanks for the email and making time to look at the proposal :D10:22
*** lachlan has joined #buildstream10:48
toscalixqinusty: commented on your changes to FAQ10:49
*** lachlan has quit IRC10:58
gitlab-br-botbuildstream: merge request (jmac/remote_execution_client->master: WIP: Remote execution client) #626 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/62611:09
tristanWSalmon, I've just finished my second reply which took a lot more time and thought11:12
tristanWSalmon, Sorry that you and Sander already replied, but I think I have a much better solution for opening multiple workspaces, I'd appreciate follow up there instead :D11:13
tristanvalentind, sorry I've been very tied up and had it on my TODO to get through this workspace proposal before it runs out of sight without me11:33
tristanvalentind, regarding your question of the install page for first time users, we should approximately follow http://buildstream.gitlab.io/buildstream/main_install.html11:34
tristanvalentind, there are some adjustments we could make; but there are essentially 3 paths to choose: Distro, Source or Docker11:35
qinustyWhere does PyPi come into that tristan?11:37
valentindtristan, ok, I proposed to move extract installation from git (or tarball) out of the installation from PyPI, but add a note box to link to the alternative method. This to avoid having user to need to hop over sections to get to the post install. Would that be OK?11:37
cs-shadowqinusty: PyPI comes under "installing via source" section11:38
qinustyI guess technically PyPI is just automating the install via source11:39
qinustypip*11:39
cs-shadowyeah, and it can't be a complete solution since we have other host dependencies11:39
tristanqinusty, `pip install BuildStream` will fail if you have not installed the native base dependencies on your distro and followed the instructions for that first11:43
tristanqinusty, since pip is unable to automatically install all of the dependencies, it is considered an installation from source indeed11:43
valentindtristan, what is the recommended way? PyPI? Or installing from distro when available?11:44
tristanvalentind, I dont understand what you have "extracted" to be honest11:44
valentindtristan, in installation from source, there are 3 sections.11:44
tristanvalentind, Let me throw up a mockup (in the middle of), and we can discuss that way11:44
valentind- dependencies11:45
valentind- installation11:45
valentind- post install11:45
valentindIn the installation section, there are 2 ways: from PyPI, or from git.11:45
valentindIf you install from PyPI, when you are done with the section, you have to hop over and continue to "post-install"11:45
valentindAnd in my opinion, it is difficult to navigate.11:46
valentindUnless we had a dynamic website where you could collapse sub sections.11:46
qinustyStatic websites can do that11:47
qinustywith JS11:47
valentindI think a tabbed box would be better than collapsing in that case.11:51
qinustyhttps://www.w3schools.com/bootstrap/bootstrap_ref_js_collapse.asp incase interested. We can div up our sections within the markdown I believe. SImilar to the way we do the id for #location11:52
tristanvalentind, Ok this is the mockup I was able to very quickly draft: https://gitlab.com/BuildStream/buildstream/snippets/175096911:52
tristanvalentind, for the main page that is, it keeps the same structure of subpages we currently have at http://buildstream.gitlab.io/buildstream/main_install.html11:52
valentindtristan, OK.11:53
valentindSo we have a sub page for the dependencies.11:53
tristanvalentind, For semantic versioning, we leave that out as it's too much information; but I think it is relevant on the "download" page, which we should link to from the "tarball" section11:53
valentindtristan, should we have a sub page for the post-install as well. I suppose so. But you forgot.11:53
*** lachlan has joined #buildstream11:54
tristanvalentind, Well, currently the dependencies are a part of the "from source" subpage, we could keep it that way11:54
tristanYes that is important too11:54
valentindqinusty, I will not have time to make some nice javascript. I am travelling tomorrow. But maybe when the site is done we can rework that.11:54
tristanvalentind, I think that the main install should guide you through the process as much as possible, by linking you to the relevant parts; but it still makes sense to have all of http://buildstream.gitlab.io/buildstream/install_source.html on one subpage11:55
tristanI.e. then you still have it all in one page if you choose to jump up and down through the content11:55
valentindtristan, OK, so one subpage for the whole source install. And links onto sections from the main install page.11:55
qinustyOkay valentind, I think it'd work with basic html classes, as long as bootstrap is already set up with the theme.11:56
mablanchjuergbi, I've updated !626 (REAPI client), would you mind having a look when you'll have time? I'm especially interested in the comments you'd have regarding the CASCache changes.11:56
tpollardcan I change the location of the buildstream cache from project.conf?11:57
juergbimablanch: sure, will do11:57
valentindtristan, OK, we go for that. But I think when we have more time we need to find a way to make navigation within the source installation sub page easier.11:57
mablanchjuergbi, Thank you!11:57
tristanvalentind, So, in order to reconcile the "Which version to use" issue for people who *want* to choose git (and those people I presume are fairly plentiful), it will be good to link to the download page (semantic versioning section of it), in the same way we link to it from the tarball section11:59
tristanI think that brings a lot of related content together well11:59
tristanAnd we leave out the artifact server install completely *for now*, it is clutter for the newcommer11:59
tristanMaybe the install guide for the artifact server should remain in the reference manual12:00
tristanvalentind, that probably makes sense since (A) We don't guarantee stability of the server side stuff from major release to release... (B) The reference documentation is a per-release thing already, bound to the version of BuildStream you have12:01
tristanand of course (C), it's more terse and involved than a newcommer experience which we want to keep as simple as possible12:01
tristantpollard, local cache location is not project data, it is user config12:04
tpollardtristan: ok, so I can do it from .config/buildstream.conf right?12:04
tristantpollard, The remote server URLs are technically only considered project data because a project might provide one to share with it's users12:04
toscalixqinusty: do you have in your radar my comments to your MR31 ?12:04
tristantpollard, however it can also be overridden from user user config12:04
tristanYes :)12:04
tristantpollard, http://buildstream.gitlab.io/buildstream/using_config.html#default-configuration12:05
tpollardcheers tristan12:05
tristantpollard, currently the user config could be better documented, while we do have full coverage; some of it is only documented via comments in the default configuration12:06
tristanmight be best to change that to match what we've done with project configuration docs, and clearly document everything in markup, while also providing the defaults at the bottom12:06
toscalixAfter lunch I will work on the Community page.12:09
toscalixthanks tiagogomes12:10
valentindtristan, OK, so I think I have fixed everything you asked for on https://gitlab.com/BuildStream/website/merge_requests/2712:27
valentindtristan, Remember I am leaving tomorrow. So if you need me to change things on this MR, please tell me today. You can finish the MR tomorrow however.12:28
*** mohan43u has quit IRC12:42
*** mohan43u has joined #buildstream12:42
WSalmonwhat's that easiest way to get build stream to build a element rather than using the cache? i have tweaked a build element and want to see what it dose, currently i just tweak the element.bst file but that seems very silly12:46
Nexusdelete the cache12:50
WSalmonumm, i gues its tells you the cache keys...12:51
* Nexus just deletes ~/.cache/buildstream/ but results may vary12:52
Nexusor remove the ref line from the .bst file, i think that works12:52
* tristan takes a look12:54
gitlab-br-botbuildstream: issue #624 ("Stack trace when log files cause UTF-8 errors") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/62413:09
tristanvalentind, semantic versioning -> download page, not FAQ, FAQ already directs user to download page where the latest versions are reflected and directly answer the "What version do I use" question13:18
valentindOK13:19
valentindtristan, toscalix wanted to put it on the portfolio.13:20
toscalixit is already there13:21
tristanErrr13:23
toscalixtiagogomes_: why the bullet points are not rendered properly?13:23
tristanSo you want to put the semantic versioning explanation in a place far away from the table which shows the release versions ?13:23
toscalixI put the explanation of the versioning where I explain the deliverables of the project13:24
toscalixonly those consuming dev snapshots care about the numbring13:24
tristantoscalix, This is quite breaking; we should see this explanation linked from the install instructions for git at the very least13:24
toscalixnumbering13:24
tristanI mean, that is where it is most relevant; user decides they want to install from git; they need an explanation of how the tags work13:25
tristanThe practical solution here; was to have semantic versioning tightly coupled with the release table on the downloads page; so that both the tarball + git instructions can point to the same place13:25
tristanIt also makes sense from the FAQ to answer "What version should I use", to a location which shows the latest versions, *and* explains the meaning of the version numbers13:26
toscalixWhich version should I use.... the lates13:26
toscalixlatest13:26
tristantoscalix, Maybe we can move it to the related downloads table; and link to it from the portfolio ?13:27
toscalixagain, the user do not care about versioning numbers. They consume releases13:27
tristantoscalix, Ummm, "The latest" is ambiguous; that is *exactly* the reason why paulsher1ood has been frustrated about this13:27
toscalixversioning numbers care for those consuming dev snapshots13:27
toscalixambiguous....I thought the badge answered that question13:27
toscalixthat question should be answered.... go to the download page and click in the badge13:28
tristantoscalix, only when you see the badge; which is on the downloads page, nicely above the list of stable releases with links to their release announcements13:28
toscalixor add the badge directly to FAQ13:28
tristantoscalix, You are only answering the FAQ part of this13:28
tristantoscalix, the *most* important place for someone to know about the semantic version, is for the power user who chose to install from git13:29
toscalixthe semantic versioning text is best placed where the different deliverables are described13:29
tristantoscalix, it makes sense to send both users who install from tarball or install from git; to the same location13:29
toscalixif you need to reference to them, add a link13:29
tristanThat location shows all the releases, their numbers, and explains the numbering13:29
toscalixbut if you do not agree....as like with the download page.... feel free to change it13:29
valentindtristan, I have put it in the download page on the MR. We can maybe decide later which one goes away.13:30
toscalixthe download page is important for users....who do not care about release numbering13:30
toscalixadding it to that page only brings irrelevant information to them13:30
tristantoscalix, That is again incorrect; users who do not care about release numbering go to the Install page, and most of them never find the download page13:31
tristanIn fact, the download page; should be renamed the "releases" page13:31
tristanIt shows all the releases, and shows which one is latest, and is rather a power user thing13:31
toscalixif it is incorrect, then fine, go ahead and change it.13:31
tristanNormal users, do not download.13:31
valentindtristan, we should think of removing the download page from the main menu and put the install page instead. Should I put that in the MR?13:31
tristanvalentind, I assumed that you did so far13:32
paulsher1ood"homonym toolset" ???13:32
tristanpaulsher1ood, eek, that looks crazy, wonder where that came from13:32
flatmushI'm currently trying to setup a buildstream cache, I followed the instructions here:13:32
flatmushhttps://buildstream.gitlab.io/buildstream/install_artifacts.html13:32
flatmushWith the only exception being that I only provide the CI with the private key for the push/pull server.13:32
flatmushBut I'm getting the following error:13:32
flatmushhttps://gitlab.com/trustable/distros/minimal-distro/-/jobs/9452505613:32
toscalixpaulsher1ood: tristan if you do not my texts, feel free to change it. I understand it is not one of my strengths13:33
toscalixdo not like ....13:33
tristantoscalix, I had to catch up with the workspaces thread before it runs off on the horizon without me today, no worries I understand13:34
paulsher1oodwhere is the source? i'll fix it13:34
tristanI have not had time to look at everything on the website in detail13:34
tristanpaulsher1ood, https://gitlab.com/BuildStream/website13:34
valentindtristan, I put the link in the menu.13:34
toscalixhttps://gitlab.com/BuildStream/website/tree/master/content13:34
paulsher1oodack13:35
qinustytoscalix, I did see your ping pop up. But can't find it in my channels... Might be blind. But yes, I'm looking into those changes as we speak13:35
tristanvalentind, Great, instead of Download I presume; since that content is now all about the upstream releases and versioning; shall we also title it "Releases" ?13:35
valentindWhat should we title release? The download page?13:36
tristanvalentind, The word "Download" (and I understand why toscalix chose it), comes with the presumption that (A) The user downloads a payload  (B) The user proceeds to install the downloaded payload;  it makes sense in many contexts13:36
valentindOK13:36
tristanBut we ended up having a nice and valuable result there13:36
tristanSo yeah, lets call it that; and we'll want to tightly couple this with the major release announcements in some way later on I suppose13:37
toscalixevery single new user will look for the word download, no matter how they consume the software13:37
* coldtom would look for install first, given i wouldn't want a tarball13:37
toscalixinstead of trying to be accurate, I would rather do the most popular thing, to be predictable13:38
* cs-shadow agrees with coldtom 13:38
tristanI understand that windows users would, and people who want to install an ISO distro image ; both would be looking for download13:38
tristanThat said, when you arrive on the site; and your choices are (A) (B) (C) and (Install), you will go to (Install)13:38
valentindtristan, done.13:39
toscalixwe are not doing these contents for "people like us"13:39
toscalixbut for a much wider audience13:39
qinustyI'd argue the people using buildstream are system integrators and software engineers.13:40
tristantoscalix, We are not going over this anymore - it's important that we dont send users to this download page when what they should be doing is following the install instructions instead13:40
qinustyIf you're installing onto a Linux system, a tarball e.g. (Download) is even more complicated to you13:40
tristantoscalix, We have to provide the critical path first13:40
toscalixwe are not developing contents for people using buildstream, but for those who do not13:40
qinustyEither way, the more complicated install route is the tarball. Hence why it should be kept for power users13:41
tristantoscalix, If we send them to a download page, they will download a tarball, and then wonder what to do with it, then they will go to the installation instructions; and find out they never needed a tarball, or to know what version they wanted in the first place13:41
toscalixthe current critical path is broken. You will need to find a new one fast or fix it later13:41
tristanI.e.: It's just a frustrating experience we can avoid.13:41
toscalixthe download page can perfectly contain the link to the right install section and a note explaining what to do.13:42
toscalixthe point is not the content but which page should be routing users after they read the release annoucnement and they read the feature page13:42
tristantoscalix, That means (A) Extra click to get where you should be... (B) Irrelevant information for the new user given to the user13:42
tristanWe want to avoid those things13:42
toscalixlast minute ideas without a big pic end up with broken design, like it happens with code.13:43
tristanUsers will not read the feature page and release announcement often, honestly; unless they are already existing users.13:43
toscalixSo I no not think is worth discussing much about it. We have too much to do13:43
toscalixI will go back to the community page13:43
tristantoscalix, Please just give up; I understand you put effort into this design, and you think there should be a download page - we agree that this is wrong and misleading especially to the new user.13:44
* toscalix wonders what a journalist will read first about a release: the announcment and then the feature page13:44
tristanA journalist is looking for something different than installing and using BuildStream13:44
tristanThey might be into the release announcements, indeed13:45
adds68flatmush, any logs on the server?13:45
toscalixtristan: many of them, the good ones, will want to install it13:45
tristanAnd we should be able to direct them there through the news feed13:45
tristantoscalix, So they will also be directed to the right places13:45
flatmushadds68: What kind of logs do you want?13:45
toscalixanyway..... my experience in that field does not seem to count either, so back to my work13:45
toscalixTristan, I see no specific license in the docu so I assume it inherits the license from the repo, which is LGPL2.1, correct?13:54
toscalixtristan ^^13:55
paulsher1oodhttps://gitlab.com/BuildStream/website/merge_requests/3313:56
toscalixpaulsher1ood: are you sure you want to name other projects in your front page? This has an impact on the searches13:57
tristantoscalix, Correct13:57
adds68flatmush, are you running it under systemd for example (the cache server)13:58
flatmushadds68: Yep13:58
flatmushas close to the example as possible13:58
adds68flatmush, is there any event logs in journalctl ?13:58
flatmushso same ports, same two unit files13:58
flatmushnothing weird happens at the point where the error occurs13:58
paulsher1oodtoscalix: why not?13:59
paulsher1oodit's the truth13:59
adds68hmm, i've seen this before where it's actually a bst issue and not your cache server13:59
tristanflatmush, this looks horrible :-/ I presume you have the latest 1.2.0, and that loading the configuration failed for some reason13:59
toscalixyou can tell the truth in another page and use this one to get a better position when searching for build tool, integration tool, etc13:59
adds68flatmush, what about dmesg?13:59
tristanflatmush, further, there should be a stack trace with a BUG message, I wonder where that went :-S14:00
tristanadds68, thanks for following up on this14:00
flatmushtristan: I'm using the 1.1.7 tag14:00
tristanflatmush, that shouldn't make a difference, I don't recall anything in configuration of caches landing between 1.1.7 and 1.2.014:00
flatmushI only gave the client-key field in the artifacts: section of the project.conf14:01
flatmushbecause I assumed that was all that's needed (otherwise I'd have to upload a load of public keys)14:01
toscalixpaulsher1ood: you can tell the truth in another page and use this one to get a better position when searching for build tool, integration tool, etc14:02
paulsher1oodmeh14:02
paulsher1oodby all means amend the commit14:03
toscalixups, approved it14:03
flatmushadds68: I'm getting an sshd key error, but I was under the impression that the cache is just http now?14:03
adds68flatmush, if you provide keys i think it is https14:04
tristanvalentind, lets go ahead with !27, it's better to land right away, observe, iterate14:04
flatmushSep 04 15:03:47 cache sshd[10657]: error: Could not load host key: /etc/ssh/ssh_host_dsa_key14:04
flatmushbut that's happening even when the test isn't running so I suspect not relevant14:04
flatmushadds68: I'm only providing the private key (client-key), because I was under the impression that the public keys aren't needed, is that correct?14:05
adds68flatmush, no they are always needed unless you are using a proper certificate authority14:08
toscalixtiagogomes: it seems that the bullet points are not rendered properly. Do I need to leave spaces in front of the buller points or something?14:08
tiagogomestoscalix where?14:08
toscalixtiagogomes: https://buildstream.build/portfolio.html14:09
tristanadds68, flatmush; As I suspect you may have found the cause of this error, can you please file an issue about what happened so we can improve the error message here ? I suspect its a very easy fix and would like to have it in 1.2.114:09
tiagogomestoscalix they look good to me :) You have to leave a blank line before starting the list14:10
toscalixtiagogomes: checking a different browser14:11
toscalixnop, they are not correct, checking sources14:11
tiagogomesWhat do you mean by "not correct"14:11
flatmushtristan: Will do14:12
toscalixtiagogomes: I see them as paragraph, not as list, on the website14:13
tristanI'm really stumped why we have a formatted exception in a BUG message without a stack trace :-S14:13
qinustyIn our message API overhaul tristan, there are a few instances of BUG messages being instantiated in different places14:14
qinustyperhaps one of them is incorrect?14:14
toscalixtiagogomes: the second paragraph should be a list in https://buildstream.build/portfolio.html14:15
tristanvalentind, Did we still not find a solution for the instructions to format like `.. code::` does ?14:15
tiagogomestoscalix you don't see this https://pasteboard.co/HCkYgL2.png14:15
toscalixah, no, wait14:15
gitlab-br-botbuildstream: merge request (willsalmon/outOfSourecBuild->master: WIP: out of sourec builds) #776 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/77614:15
qinustytristan, I think it's a css thing. ```code``` is definitely a code block14:15
tristanvalentind, I.e. nicely boxed code examples, not bold burgundy text14:15
tristanqinusty, Hmmm, ok so I guess we need to fix that in the theme ?14:16
qinustyperhaps14:16
tristanLooks really bad the way it is14:16
qinustyit's putting it inside a <pre> block14:16
qinustyI agree14:16
tristantiagogomes, are you on it ?14:16
qinustyhttps://buildstream.gitlab.io/buildstream/HACKING.html#feature-additions14:16
qinustyrip wrong link14:16
toscalixtiagogomes: I see it, correct, the lists are there, it is just that there are asterisks in the middle of the text. Checking why14:16
qinustyhttp://docs.getpelican.com/en/3.6.3/content.html#syntax-highlighting14:16
qinusty"For Markdown, which utilizes the CodeHilite extension to provide syntax highlighting, include the language identifier just above the code block, indenting both the identifier and the code:"14:16
qinustyThey aren't using ```14:17
tiagogomestoscalix there are asterisks because you put them there in the source14:18
*** lachlan has quit IRC14:18
toscalixtiagogomes: my first report is correct. Some of the lists in that page are not rendered properly14:18
tristanvalentind, ^^^^^^ See what qinusty said here regarding how to format the shell commands people need to copy paste while installing14:18
qinustyIt looks like indented blocks are code blocks to pelican?14:18
tristanvalentind, I suppose we need to use ``` (like what we use in gitlab markdown, actually)14:18
toscalixtiagogomes: maybe it is because there should be a line between text and lists14:19
tristanThen we do ```python or ```yaml if we want syntax highlighting for those14:19
tiagogomes<tiagogomes> toscalix they look good to me :) You have to leave a blank line before starting the list14:19
toscalixyour assumption was correct. It renders properly in gitlab but not in pages14:19
gitlab-br-botbuildstream: issue #625 ("BUG without stack trace when public keys are not specified for an artifact cache using a self-signed certificate") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/62514:21
WSalmonI had a chat with juergbi about "Out of source builds" and we came up with https://gitlab.com/BuildStream/buildstream/merge_requests/776/ i think we have got to quite a nice place but before i spend ages cleaning it up i would really appriciate some architectureal feed back on if i have gont too far especally ragards to  buildstream/buildelement.py and buildstream/data/projectconfig.yaml many thanks (tristan) i know you are mega busy but if you14:24
WSalmonare going to veto this is would be great to get it earlier than later :)14:24
paulsher1oodtoscalix: sorry, i introduced a typo... https://gitlab.com/BuildStream/website/merge_requests/3714:24
tiagogomesAesthetically, what bothers people most on the website14:27
tristanWSalmon, sigh... ok it's on my desk first thing tomorrow, I hope it's not changing too much stuff14:27
valentindtristan, We use ```14:28
tristanWSalmon, my expectation was this was all doable from yaml for individual elements which support that; my thoughts are the core should not know anything about out of source builds14:28
tristanWSalmon, those are only a detail that some build elements can implement in their yaml (not all of them can do it, even)14:29
qinustyvalentind, their docs use indentation for code blocks14:29
valentindOh so codehilite?14:29
qinustyhttp://docs.getpelican.com/en/3.6.3/content.html#syntax-highlighting14:30
qinustyvalentind ^14:30
tristanMissing a plugin ?14:30
qinustySyntax for plugin highlighting, but the actual code block example for markdown14:30
qinustyis just indented.14:30
paulsher1oodtiagogomes: shall i attempt to fix the 'first bullet point is too high' problem?14:31
tristansounds familiar, so many flavors of markdown14:31
tiagogomespaulsher1ood go for it14:31
WSalmontristan, it is almost all in yaml 3 lines in python, i hope its ok but if you dont like the core bits than it still all works its just a bit more long hand, thanks for finding some time tomorrow :D14:32
tristanWSalmon, I'll do my best yeah; I'll take a look; about to hit the road now it's 11:3014:33
WSalmonthanks :D14:34
tristanvalentind, tiagogomes; by the way; when we run `make publish` https://gitlab.com/BuildStream/website/blob/master/Makefile ... I don't understand why, but pelican seems to be reporting success even when it spews errors in the output14:35
tristanWould be good to fix it so that we don't accidentally publish broken changes14:36
tiagogomesAre really errors rather than warnings?14:36
gitlab-br-botbuildstream: merge request (mac_fixes->master: WIP: Resolve "os.sched_getaffinity() not supported on MacOSX Blocks #411") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/72614:37
tristantiagogomes, it happened with: "ERROR: Could not process pages/download.md"14:37
tristantiagogomes, with a result of a missing page, also the accompanying warning was pretty bad14:37
tiagogomesThat doesn't look like a warning :o14:37
tristanIt would be good to error out on warnings also, we do that for the sphinx builds and it helps catch all sorts of stuff, like broken internal links14:38
tristananyway, it's since been fixed; but the fact that we don't catch the errors is bad :)14:39
tiagogomesThere's nothing on the CI configuration or Makefile that could cause that. I can only think that pelican does not return a non zero exit code on such errors14:40
*** tiagogomes has quit IRC14:40
*** tiagogomes has joined #buildstream14:41
tiagogomesThere's nothing on the CI configuration or Makefile that could cause that. I can only think that pelican does not return a non zero exit code on such errors14:42
valentindtristan, Using codehilite: https://gitlab.com/BuildStream/website/merge_requests/3814:43
valentindHowever there is no box around. But we can change that in the CSS.14:44
tristantiagogomes, I just noticed we dont run pre-merge CI14:45
flatmushtoscalix: jjardon: Can I get developer access to the buildstream gitlab? I want to push a branch which updates to the artifact cache document.14:45
tristantiagogomes, in BuildStream, we *build* the docs in pre-merge and post-merge, and it creates an artifact; then we *only* publish the docs in a separate step when running from master14:45
toscalixflatmush: on it14:45
tristanhttps://gitlab.com/BuildStream/buildstream/blob/master/.gitlab-ci.yml#L14014:46
tristanhttps://gitlab.com/BuildStream/buildstream/blob/master/.gitlab-ci.yml#L23014:46
tristantiagogomes, ^^^ that might help14:46
tiagogomespelican always returns 0 :(14:46
tristanwtf ?14:46
tristanAlways ? not even an option ?14:47
toscalixflatmush: done14:47
tristanDo they provide a linter instead or smth ?14:47
flatmushtoscalix: thanks14:47
tristanDoes anyone even use pelican, then ?14:47
tiagogomes`  --fatal errors|warnings`14:47
tristanAh, that would do it right ?14:47
valentindI have noticed that in plugins when you raise an exception, it is just swallowed and it continues.14:48
tiagogomesyeah, it works. Strange it is not on by default for errors14:48
tristancool14:48
tristanI mean, weird for plugins, but at least we can make these things fatal and CI the site for breakage14:48
* tristan hits the road... enough ! hehe14:49
gitlab-br-botbuildstream: merge request (benbrewer/install-artifacts-doc-improvements->master: Improve documentation for artifact cache installation) #777 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/77714:50
flatmushadds68: Can you have a look at that merge request ^^14:51
*** tristan has quit IRC14:52
*** tiagogomes has quit IRC14:57
*** tiagogomes has joined #buildstream14:58
gitlab-br-botbuildstream: merge request (benbrewer/install-artifacts-doc-improvements->master: Improve documentation for artifact cache installation) #777 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/77714:58
qinustyhttps://gitlab.com/BuildStream/website/merge_requests/31#note_98750482 toscalix, can you elaborate?15:04
WSalmonflatmush, what dose `_ do?15:07
WSalmonoh15:07
WSalmoni hadnt seen the first `15:08
WSalmondose it make the <> a link?15:08
*** lachlan has joined #buildstream15:08
flatmushWSalmon: Good question, I copied it from another page, seemed to be the way external URL's are linked.15:08
flatmushThe stuff within the weird quotes is the text, and the angle brackets contain the link, not sure why there has to be an underscore at the end, but it's used that way in other files15:09
toscalixqinusty: canoot now. I do not remember the context. I resolved the discussion15:09
WSalmonhttps://thomas-cokelaer.info/tutorials/sphinx/rest_syntax.html seem so15:10
gitlab-br-botbuildstream: merge request (mac_fixes->master: WIP: Resolve "os.sched_getaffinity() not supported on MacOSX Blocks #411") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/72615:11
*** lachlan has quit IRC15:13
*** tristan has joined #buildstream15:16
*** ChanServ sets mode: +o tristan15:16
gitlab-br-botbuildstream: merge request (mac_fixes->master: WIP: Resolve "os.sched_getaffinity() not supported on MacOSX Blocks #411") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/72615:17
gitlab-br-botbuildstream: merge request (willsalmon/outOfSourecBuild->master: WIP: out of sourec builds) #776 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/77615:18
toscalixtiagogomes: https://gitlab.com/BuildStream/website/merge_requests/36/15:22
gitlab-br-botbuildstream: merge request (mac_fixes->master: WIP: Resolve "os.sched_getaffinity() not supported on MacOSX Blocks #411") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/72615:23
tiagogomestoscalix that has merge conflicts15:26
toscalixah ok15:29
WSalmonany one got a good cmake example i can use to test my out of source builds with?15:32
WSalmoncoldtom, adds68 maybe ^15:32
coldtomthere's a few cmake elements in freedesktop, fcitx is one i had to fix recently iirc15:34
WSalmonta15:36
gitlab-br-botbuildstream: merge request (mac_fixes->master: WIP: Resolve "os.sched_getaffinity() not supported on MacOSX Blocks #411") #726 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/72615:44
*** lachlan has joined #buildstream15:48
toscalixtiagogomes: I think now is better https://gitlab.com/BuildStream/website/merge_requests/3915:48
toscalixah, so the name of the download page changed .... can we put it back please? Otherwise we already have plenty of broken links all over the place15:49
juergbiKinnison: regarding #583, so .close() on Queue and Popen objects don't actually close all fds, or are we failing to invoke close in appropriate places?15:51
Kinnisonjuergbi: It's insufficient15:51
toscalixvalentind: change it back, please15:51
Kinnisonjuergbi: I believe I have a fix15:51
Kinnisonjuergbi: I'm just testing it15:51
juergbi:)15:51
* Kinnison stashes his scary monkeypatching of os.pipe() and os.close()15:51
tiagogomestoscalix what do you mean?15:52
toscalixdownload.html does not exist anumore15:53
toscalixanymore15:53
*** dtf has joined #buildstream15:54
KinnisonDoes anyone know why I might get https://pastebin.com/KJB4jXWk when trying to do 'python3 setup.py test' ?15:54
*** lachlan has quit IRC15:55
toscalixwrote a mail to tristan and valentind so they revert the name change of the page or fix the links in the rest of the pages15:55
valentindtoscalix, I will fix it.16:00
toscalixthanks16:00
valentindhttps://gitlab.com/BuildStream/website/merge_requests/4016:03
tpollardnot pulling in the buildtree of llvm feels so liberating16:07
tlater[m]Kinnison: WIld guess, but we've had issues with pylint and pylint-runner version mismatches...16:13
tlater[m]There's a chance you're using incompatible versions - I don't think we've gotten around to pinning the version yet.16:13
KinnisonI dumped my .eggs and it is running now16:13
Kinnisonthanks16:14
tlater[m]Ah, alright :)16:15
toscalixqinusty: for now, let's add it as planned in the website. If we move it, it will be in the future16:15
KinnisonOkay, that's fun, loads of lint errors for code which wasn't touched by me /o\16:15
Kinnisone.g.16:15
Kinnison_____________________________________________________________ [pylint] buildstream/sandbox/_sandboxbwrap.py ______________________________________________________________16:15
KinnisonR:142,12: Consider using dict.get for getting values from a dict if a key is present or a default if not (consider-using-get)16:15
Kinnisonshould I ignore those?16:15
tiagogomestoscalix what remains to be done in the website16:16
tlater[m]That in turn is known to be caused by newer versions of pylint than we're used to - qinusty would know which errors that brings up exactly.16:16
Kinnisonokies, so I can ignore them.  Cool16:16
* Kinnison might (terrifyingly) have an MR ready soon16:16
tlater[m]Kinnison: #570 tells you exactly what errors you can ignore, but that one seems safe :)16:17
*** solid_black__ has quit IRC16:18
gitlab-br-botbuildstream: merge request (tiagogomes/issue-573->master: Reduce IO overhead caused by artifact cache size calculation) #671 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/67116:20
KinnisonWhat should the approvers list look like for an MR?  It seems to default to requiring tristan approve16:23
qinustyDepends on the impact of the MR Kinnison16:25
* tlater[m] isn't sure if "a few who seem qualified" is good enough anymore. Happy to review if you need someone though16:25
Kinnisontlater[m]: does it cause issues if I leave it set at tristan?16:26
qinustyIf it's fairly trivial, you can have one or two contributors review it16:26
* Kinnison knows tristan weighed in on the issue (#583)16:26
tlater[m]No, that shouldn't be an issue16:26
qinustyNah, it can be left. MR's have a requirement of 0 reviewers technically16:26
KinnisonOkay, I'll leave it as is16:26
gitlab-br-botbuildstream: merge request (danielsilverstone-ct/maybe-reduce-fd-leaks->master: jobs.py: Reduce FD leaks from queues and process objects) #778 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/77816:26
* qinusty will take a look once you push it16:26
Kinnisonqinusty: https://gitlab.com/BuildStream/buildstream/merge_requests/77816:27
* tlater[m] sees Kinnison is playing with fire16:28
Kinnison?16:28
tlater[m]Oh, nevermind, it's just a message queue16:28
* Kinnison would be quite worried if the queue was on fire16:29
toscalixtiagogomes: still aorund?16:30
qinustyKinnison, how does this behave when retries are involved? I am not familiar with the use of a multiprocessing queue between multiple forked processes16:30
*** lachlan has joined #buildstream16:31
tiagogomestoscalix yes16:31
Kinnisonqinusty: unless I'm very mistaken, if the job needed to retry, it'd rerun Job.spawn() which would recreate the queue and process instances it needed16:31
toscalixtiagogomes: https://buildstream.build/community.html16:31
* Kinnison has a quick peek16:31
*** dtf has joined #buildstream16:32
qinustyKinnison, ignore me16:32
qinustyI am being tired and reading the diff back to front16:32
Kinnisonhehe16:32
toscalixtiagogomes: I should assume you need 6 spaces for a subitem in a list16:32
toscalixbefore the asterisk16:32
toscalixcomparing the web page with the source: https://gitlab.com/BuildStream/website/blob/master/content/pages/community.md16:33
tiagogomespossibly, you know you can test the changes locally easy16:33
toscalixhow?16:33
gitlab-br-botbuildstream: merge request (danielsilverstone-ct/maybe-reduce-fd-leaks->master: jobs.py: Reduce FD leaks from queues and process objects) #778 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/77816:34
toscalixwhen I transfer them to html, the rendering is different from the web16:34
qinustyLooks good to me Kinnison16:34
toscalixI export the .md to html16:34
Kinnisonqinusty: thanks for the vote of confidence16:34
tiagogomestoscalix it is on the README file16:34
qinustyJust to note, Remember to click `Remove Source Branch` when creating an MR in future :D16:34
qinustyI've ticked it when the merge occurs16:34
tlater[m]Does `del` actually cause python to remove the objects?16:35
*** lachlan has quit IRC16:35
tlater[m]Or does it simply mark them as unused, meaning you have to wait for the garbage collection anyway?16:35
Kinnisontlater[m]: I think it both removes the reference and tries to destroy16:35
toscalixtiagogomes: thanks16:36
Kinnisontlater[m]: certainly it seems to keep the total open FDs below about 25 on my test case16:36
qinustyIt calls __del__ tlater, which is a destructor16:36
Kinnisontlater[m]: where without my patch it's up at around 40,000 and rises continuously16:36
* tlater[m] double checks16:36
tlater[m]For some reason I have in my mind that setting to None is actually equivalent16:36
tlater[m]Oh, well, I won't argue with the test then16:36
qinustyI mean, that's a slight reduction Kinnison16:36
tlater[m]Just a tiny one16:37
tlater[m];p16:37
Kinnisonqinusty: It was a bit of a torture test, to be fair16:37
qinustyStill16:37
tlater[m]I suppose there's no easy way to include such a test in the test suite?16:37
Kinnisontlater[m]: probably not without monkeypatching os.pipe() and os.close() which I did originally to work out where we were leaking the FDs from16:38
gitlab-br-botbuildstream: merge request (danielsilverstone-ct/maybe-reduce-fd-leaks->master: jobs.py: Reduce FD leaks from queues and process objects) #778 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/77816:39
tlater[m]Thanks for figuring that out then :) Eventually we really should get the performance tests working, so that we don't miss details like these anymore.16:40
Kinnisontlater[m]: It was an amusing debugging effort.  At one point I thought multiprocessing.popen_fork was leaking16:41
* Kinnison was very pleased he didn't end up needing to file a bug against Python itself16:41
qinustyBut yeah, Nice fix Kinnison :D I'd never come across the issue myself16:41
gitlab-br-botbuildstream: merge request (benbrewer/install-artifacts-doc-improvements->master: Improve documentation for artifact cache installation) #777 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/77716:43
*** WSalmon_ has joined #buildstream16:45
Kinnisontlater[m]: Given the nit you raised, I'll doublecheck if =None is sufficient in this context16:45
*** WSalmon has quit IRC16:46
tlater[m]Hmm... I am thinking about retries now that I've read chat here.16:46
*** toscalix has quit IRC16:46
tlater[m]iirc jobs are pushed back to the top of the queue when they are retried16:47
tlater[m]I'm not sure if spawn() is called again16:47
KinnisonAFAICT they'd need to be, in order to create a new subprocess to run16:47
KinnisonIt looks like the dels are not necessary16:47
tlater[m]Ah, right, we do call  Process() in spawn16:48
Kinnisontlater[m]: assuming retries are tested, we'd know soon enough16:48
* Kinnison 's test is showing the `del`s aren't needed, so I'll push a cleanup without them16:49
tlater[m]Retries are not tested to my knowledge, since they are part of the interactive features16:49
tlater[m]Which remain entirely untested... Hence I'm a bit careful about this16:49
Kinnisonaah16:49
tlater[m]That said, reading back into the code paths here I think it's fine16:49
gitlab-br-botbuildstream: merge request (danielsilverstone-ct/maybe-reduce-fd-leaks->master: jobs.py: Reduce FD leaks from queues and process objects) #778 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/77816:51
tlater[m]Between qinusty and me I think it's reviewed enough, feel free to merge after that fix, Kinnison :)16:51
KinnisonI can't16:51
Kinnison'tis my first ever buildstream commit16:51
tlater[m]Oh, right16:51
KinnisonI've updated the MR with `del` removed16:51
tlater[m]I'll merge it then, permissions for the second patch, iirc?16:52
* Kinnison is in no hurry to be granted merge rights :-)16:52
* Kinnison doesn't commonly program in python, so extra eyes are always appreciated16:53
tlater[m]Heh, merge rights are mostly so I don't need to rebase your patch every time someone else pushes something16:54
tlater[m]... which, of course, just happened16:54
Kinnisonhehe16:54
KinnisonI'm in no rush, I'm about to run away for the evening anyway :-)16:55
tlater[m]I'll still get it done, better than having it lie bare for months16:56
tlater[m]o/ Kinnison16:56
gitlab-br-botbuildstream: merge request (danielsilverstone-ct/maybe-reduce-fd-leaks->master: jobs.py: Reduce FD leaks from queues and process objects) #778 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/77816:57
Kinnisonthanks tlater[m]16:57
*** jonathanmaw_ has quit IRC17:01
gitlab-br-botbuildstream: merge request (danielsilverstone-ct/maybe-reduce-fd-leaks->master: jobs.py: Reduce FD leaks from queues and process objects) #778 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/77817:20
*** lachlan has joined #buildstream17:24
*** rdale has quit IRC17:27
*** lachlan has quit IRC17:41
*** leopi has quit IRC18:32
*** leopi has joined #buildstream18:43
*** leopi has quit IRC18:46
* tristan is pulling freedesktop-sdk.bst:extensions/rust/rust.bst for an hour and a half now18:49
tristanat like ~3K per second18:49
coldtomtristan: it /is/ big but i think that's a little excessive nonetheless18:49
* tristan gives a cold stare to https://gitlab.com/BuildStream/buildstream/issues/55418:50
tristangive me a stream, dont round trip and download a gazillion tiny files one by one18:50
*** bochecha has quit IRC18:52
*** leopi has joined #buildstream18:53
*** leopi has quit IRC19:23
coldtomhas anyone managed to properly install and configure qt5 using buildstream? I can't get it to both install to an install root and be configured correctly19:29
coldtommore specifically, it seems that installing to an install root requires you to include it in the prefix argument passed to configure, but doing that means it's preserved in qmake configuration after it's been integrated and moved into the correct place in the artifact19:30
tristancoldtom, that's the ugly that I remember indeed; you gotta build qmake into a prefix; and then everything is hard coded to what that qmake thinks is true19:44
tristancoldtom, I could be wrong, but that's how I recall it was used in baserock19:44
coldtomtristan: thanks, that's less simple than i thought this would be :P19:49
*** tristan has quit IRC20:33
*** tristan has joined #buildstream20:35
*** alatiera_ has quit IRC21:09
*** tristan has quit IRC21:51
*** alatiera_ has joined #buildstream21:58
*** alatiera_ has quit IRC22:09

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!