IRC logs for #buildstream for Wednesday, 2018-03-14

*** mcatanzaro has joined #buildstream00:16
*** mcatanzaro has quit IRC03:04
*** Prince781 has joined #buildstream03:23
*** Prince781 has quit IRC04:31
*** valentind has joined #buildstream06:57
*** valentind has quit IRC07:22
*** dominic has joined #buildstream08:30
*** dominic has quit IRC08:31
*** juergbi has quit IRC08:49
*** juergbi has joined #buildstream08:53
*** toscalix has joined #buildstream08:55
*** tristan has quit IRC09:19
*** tristan has joined #buildstream09:23
*** dominic has joined #buildstream09:24
tlaterAh, juergbi, about classmethods, the `cls` there refers to the class that this function is part of09:38
tlaterI.e., if you inherit from a class that implements one of these class methods09:38
tlaterYour new class will be able to refer to itself through `cls`09:38
juergbii'm missing the context09:39
juergbiwhen did we talk about classmethods?09:39
tlaterRather than pointing to its parent, which you'd usually get if you referred to it directly09:39
tlaterjuergbi: Yesterday, I was a bit late to this x)09:39
juergbidid you mean jennis/jmac?09:39
tlaterOh, sorry, yep09:39
tlaterjennis in particular - silly autocomplete finger09:40
*** jonathanmaw has joined #buildstream09:42
*** aday has joined #buildstream09:50
jennisahh thanks tlater09:56
jennisis @classmethod almost like a builtin decorator then?09:57
tlaterjennis: It *is* a builting decorator :)09:57
tlaters/ing/in/09:57
jennisSo it looks to me as if we're replacing the ScriptWriter.get_args() function with our own slightly modified version?10:03
jennisScriptWriter get_args(): https://github.com/pypa/setuptools/blob/master/setuptools/command/easy_install.py#L209310:03
jennisours: https://gitlab.com/BuildStream/buildstream/blob/master/setup.py#L16510:03
tlaterjennis: Yep, that's exactly what's happening10:06
* tlater assumes this came up because pylint can't figure this out and is giving you a warning10:06
tristanjmac, have a moment ?10:30
jmacNot during standup10:30
tristanjmac, :)10:30
jmac5 mins10:30
*** aday has quit IRC10:31
tristanAnyway I'll write...10:31
tristanjmac, so; regarding the sequence ID thing... I can understand that; in order to get benchmarking "off the ground" there will be some churn, which is fine and acceptable especially during unstable dev cycle10:32
tristanI am only a little concerned about the long term picture; the idea of sequence ID for a task, and another for a subtask seems very sensible; and while it may seem to be prematurely over-engineering a thing... in this case I just feel that some foresight can pay off10:34
tristanI just wanted to say a word about this since you are looking into it... the reason being; for a benchmarking suite; it is interesting to remain backward compatible; so that we can compare performance far back into history later on10:35
jmacI agree that main activtiy/subactivity ids are the right way to go10:35
jmacI'm not actively looking into it at the moment though10:36
tristanI can imagine say; that benchmarking will eventually "grow new things" which are not comparable with earlier versions of buildstream which dont time certain things10:36
tristanbut it would be nice that the things we can benchmark from the beginning, are benchmarkable forever10:36
tristanjmac, understood :D10:36
jmacThat would definitely be desirable, and that's why also codifying the activity names is a good plan10:37
jmacAt the moment we don't really know what we want out of our benchmarks, so I'm throwing possible formats around10:38
*** aday has joined #buildstream10:38
tristanjmac, sure - I just wanted to point out that... in this scenario, a little over-engineering can have long term benefits :)10:39
tristanI think we're on the same page, though10:39
tristan(it's been sitting in the back of my mind for a while, just wanted to communicate it)10:40
jmacThank you10:40
Nexustristan, tlater i ran the last command that it seems to run (printing out the args and turning that into a bst command)10:42
Nexusand it works10:42
tlaterAh, interesting10:42
Nexusthe build succeeded10:42
tlaterNexus: Have you checked what the test that runs *just before* our hanging test does?10:42
* tlater suspects it might be messing with the environment10:43
tlaterI.e., try perhaps only running the hanging test with the test suite10:43
Nexusi have the build log on my screen if thats what you're talking about10:43
tlaterHm, no, not quite10:43
tlaterYou know when you run the whole suite with `sudo BST_FORCE_BACKEND=unix ./setup.py test --integration`?10:44
* Nexus nods10:44
tlaterEventually you end up with the hanging test we're trying to debug10:44
tlaterWhat if the test that runs *just before* that test screws up everything?10:44
Nexuswell then we're doomed10:45
NexusDOOOOOMED10:45
Nexusi'll look into it10:45
tristan ./setup.py test --addargs 'path/of/failing/test::test_name' is interesting, so you know if it works when you *just run that test*10:45
tlaterNexus: It might be worth running: `sudo BST_FORCE_BACKEND=unix ./setup.py test --addopts '--integration tests/<whatever test is hanging>'`10:45
tristanright, like tlater says :)10:45
* tlater seriously needs to write a little doc for these test invocations10:46
*** aday has quit IRC10:51
Nexusseems to hang [tristan tlater]10:52
gitlab-br-botbuildstream: merge request (jmac/composition-docs-fix->master: Docs: Better explanation of defaults) #316 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/31610:52
tlater:(10:52
tlaterIt's only running that test and no others?10:52
Nexusyup10:52
*** aday has joined #buildstream10:55
tristanjuergbi, Say I have a loaded build graph... and there "are junctions in play", given an element in the graph: Can I get a list of the closest dependencies which cross a junction boundary ?10:58
tristanthis would be very helpful in constructing a useful error message10:59
juergbitristan: i don't think there is no code to determine that yet10:59
tristanI think you mean the opposite of what you said :D11:00
tristanbut yeah, I suspected that much, just curious11:00
* tristan wonders if we have something efficient for this...11:00
tristanElement.dependencies() could be breadth first if recurse=False11:01
tristanor not necessarily even breadth first, just doing non-recursive loop could let me do it11:01
tristanjuergbi, also, what would you use to detect that ? ... an element can *depend on an element across a junction boundary* as I understand it11:03
tristanjuergbi, so what we have currently, is Plugin._get_project() + Project._junction ?11:04
tristanmaybe a private direct accessor on the Element class would be nice ?11:04
tristanor maybe for this case, I just want to check `if dependency._get_project() is not search._get_project()`11:05
juergbitristan: you can anyway directly compare project instances11:10
juergbino need to compare the junction name11:10
gitlab-br-botbuildstream: merge request (jmac/composition-docs-fix->master: Docs: Better explanation of defaults) #316 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/31611:12
*** aday has quit IRC11:15
*** aday has joined #buildstream11:23
tristanjuergbi, yup, on it... I'm a bit slow today... went to dentist... scary11:33
tristanjuergbi, at what point in _pipeline.py do I know that the pipeline is fully loaded ? I'm not seeing where junctions get added11:34
tristanAh11:34
tristanjuergbi, nice, you do it *all* in _loader.py it seems, that's convenient :)11:35
* tristan starting to get the feel of junctions in the codebase11:35
tristanlooks like it might be nice to shove all Project() instantiation into Loader()11:37
* tristan leaves things as is, though11:37
juergbii.e., move toplevel project creation to Loader? would mean CLI options have to be passed through somehow but it could make sense11:39
tristanJust seeing as the Loader() now creates projects... things are getting "biggish", would be desirable to streamline more11:40
tristannot sure exactly it makes the most sense11:40
juergbimay be worth looking into, yes. top-level project is currently created very early and not in a single place, so would be a slightly wider change11:42
tristanOk this is weird, I should fix this...11:49
tristanCLI -> App -> Pipeline: Initialization is confusing with regards to elements, except_, and track_elements11:50
tristanreally; so what is except_ when you initialize a pipeline now ?11:50
*** aday has quit IRC12:04
*** aday has joined #buildstream12:07
tristantlater, jennis... just curiously; does your linting work cover mutable default function arguments ?12:30
tlatertristan: Yes!12:31
tristanNice :)12:31
* tlater thinks jennis also refactored a few of the ones that sneaked into the repo12:31
* tristan just discovered except_=tuple() in main.py12:31
tristanprobably one of them12:31
tristanthey are easy to forget12:32
juergbithose are actually still in the pylint branch, hm12:36
juergbituples are immutable and thus it's not a dangerous default value?12:36
tristanAh, I didnt realize they are immutable12:37
tristannot mutable, not dangerous12:37
jennisYes it didn't come up through the linting, many default assignments such as foo=[] were changed12:40
jennisSo I suspect juergbi is right12:40
tristanhttps://docs.python.org/3/tutorial/datastructures.html#tuples-and-sequences12:45
tristanbelow example: "Tuples are immutable, and usually contain a heterogeneous sequence of elements that ... bla bla"12:45
gitlab-br-botbuildstream: merge request (239-use-pylint-for-linting->master: Resolve "Use pylint for linting") #283 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/28312:48
jennisjuergbi, I've pushed the branch ^12:48
tristanjuergbi, we may need an overhaul of where element.name is used, but element._get_full_name() should be used instead12:49
jennisplease could you review it12:49
jennisTristan, we also have a element.normal_name variable12:50
tristanjennis, I think we recently decided basically that that should be trashed12:51
juergbiyes, should revisit that12:51
juergbijennis: ta, will take a look12:51
*** mcatanzaro has joined #buildstream12:52
* tristan takes a short list of things he wants to get to refactoring pronto12:52
juergbijennis: has pylint problems dealing with += or what was that about?12:56
tlaterjuergbi: Pylint uses heuristics to try and guess the type of some variables12:58
tlaterThose heuristics fail in that specific location...12:59
juergbilooks like quite an odd failure but wel12:59
juergbi*well12:59
tlaterjennis said he'd raise an issue on their repo about it12:59
juergbithat sounds good12:59
juergbiwhat happens inside list comprehension should be unrelated to the thing outside list comprehension (except for used variables)13:00
tlaterPylint occasionally tries to guess types for variables based on how you use them - if it can't find the corresponding class definition13:01
tlaterI think it's something about += to a string13:01
juergbii would prefer conservative heuristic13:01
juergbifalse positives are bad for warning tools like that, imo13:02
* tlater agrees13:02
tlaterI think we can set the "confidence" level, but I'm not sure how much it actually affects13:02
tlaterjennis: Do you know if we can blanket disable type guessing?13:02
* tlater doesn't quite remember what the config allowed13:03
juergbiah. the situation is not really bad right now, i'm not opposed to keeping it as is. i was just wondering13:03
juergbi(i'm also not opposed to further tuning, of course, but can also do this later)13:04
tlaterYeah, the config is pretty well-tuned at this point :)13:04
tlaterIs it just me or are the (non-integration) tests much slower these days?13:06
gitlab-br-botbuildstream: merge request (215-workspace-builds-might-not-rebuild-correctly-when-dependecies-are-updated->master: WIP: Resolve "Workspace builds might not rebuild correctly when dependecies are updated") #315 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/31513:11
gitlab-br-botbuildstream: merge request (215-workspace-builds-might-not-rebuild-correctly-when-dependecies-are-updated->master: WIP: Resolve "Workspace builds might not rebuild correctly when dependecies are updated") #315 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/31513:12
gitlab-br-botbuildstream: merge request (215-workspace-builds-might-not-rebuild-correctly-when-dependecies-are-updated->master: WIP: Resolve "Workspace builds might not rebuild correctly when dependecies are updated") #315 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/31513:12
gitlab-br-botbuildstream: merge request (215-workspace-builds-might-not-rebuild-correctly-when-dependecies-are-updated->master: WIP: Resolve "Workspace builds might not rebuild correctly when dependecies are updated") #315 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/31513:15
tristantlater, I think there are more of them13:19
tlaterThat does make sense13:20
juergbihm, duplicated comment in gitlab :/13:22
*** tristan has quit IRC13:24
tlaterGah, looks like gitlab isn't running CI properly atm13:24
tlater10 minutes to start pulling a docker image :|13:25
*** tristan has joined #buildstream13:28
gitlab-br-botbuildstream: merge request (215-workspace-builds-might-not-rebuild-correctly-when-dependecies-are-updated->master: WIP: Resolve "Workspace builds might not rebuild correctly when dependecies are updated") #315 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/31513:45
jennistlater I'm not sure off the top of my head14:03
jennisWould've thought that whereever you read this: <tlater> Pylint occasionally tries to guess types for variables based on how you use them - if it can't find the corresponding class definition.   It would have said14:04
jennisassuming you read that somewhere ;P14:04
gitlab-br-botbuildstream: merge request (239-use-pylint-for-linting->master: Resolve "Use pylint for linting") #283 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/28314:08
tlaterjennis: That was a blog post iirc, they didn't mention how to disable this.14:10
NexusIs anyone against the use of `arpy` for dealing with the dpkg plugin?14:36
gitlab-br-botbuildstream: merge request (214-filter-workspacing->master: Make workspace commands on a filter element transparently pass through to their build-dependency) #317 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/31714:47
gitlab-br-botbuildstream: merge request (issue-243_dpkg_import_source_plugin->master: WIP: Created DPKG Source plugin for Issue #10) #305 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/30514:57
jmacCan anyone point me at the bits in the code which cause element keys to inherit from project config?15:04
gitlab-br-botbuildstream: issue #214 ("We need a way to make elements depend on a subset of an element's output") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/21415:06
jmacOk, think I found it, n/m15:13
gitlab-br-botbuildstream: merge request (change_the_location_of_generate-base.sh->master: Changed the location of generate-base.sh) #307 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/30715:19
gitlab-br-botbuildstream: merge request (214-filter-workspacing->master: Make workspace commands on a filter element transparently pass through to their build-dependency) #317 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/31715:25
ltujennis, how come https://gitlab.com/BuildStream/buildstream/merge_requests/307 was closed?15:29
ltui wonder if we should make a note on the MR for provenance15:29
jennisIt was just a very small change that juerg didn't think was necessary15:29
jennisjuergbi, at the top of the cli file I disabled redefined-outer-name warnings for the whole file. I think I did this before because I didn't realise that you also had to modify the arguments of the click decorator (otherwise it fails most of the tests)15:38
jennisSo I've disabled it and will change the "track", "build" and "workspace" so we're consistent15:39
jennisunless you disagree?15:39
*** valentind has joined #buildstream15:57
milloniis Nexus around?16:01
Nexusmilloni: o/16:02
milloniNexus, could you please do git branch --contains 10fa1b4ec2ac5205230397b63d5652d7db2bb26816:02
millonion your fork of buildstream16:02
Nexusof modandtest?16:03
milloniit will check all branches16:03
Nexusno such commit 10fa1b4ec2ac5205230397b63d5652d7db2bb26816:04
milloniok16:04
millonii could not build the docs on modAndTest16:04
millonibut I could do that on that branch rebased off of master on upstream16:04
Nexushmm16:05
Nexusah, thats from the main repo16:06
Nexusnot my fork16:06
milloniNexus, were you building docs from the fork or main repo?16:08
Nexusfork16:08
milloniok16:08
milloniit mightve changed anyway commit shas actually16:08
millonianyway it builds if you include all commits from master16:09
milloniso that's that16:09
Nexusi just built the docs in a containter, and they still build needing only sphinx and sphinxclick16:09
millonibut sphinx parses the source code of buildstream doesn't it?16:09
millonithe error is when parsing the source code, not the docs that you've written16:10
Nexusthis issue is that your docs are referencing code that does not exist16:10
milloniso the source code from the snapshot on modAndTest doesn't parse well, but updated source code from master does16:10
Nexustlater, jonathanmaw have either of you got any thoughts on this?16:11
millonithese are autogenerated from .rtf from the source code16:11
millonithey're not written by a human16:11
Nexuswhat was the error you got again?16:12
jennisThese two functions `msg_byteorder` and `sys_byteorder` both have eachother as arguments. Is this intentional? Doesn't seem right to me16:17
jennishttps://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_artifactcache/pushreceive.py#L6316:17
milloniNexus, https://paste.codethink.co.uk/?418116:18
juergbijennis: it looks indeed a bit odd16:21
jennisI've changed it so that the arguments do not have the other function named and running the tests now16:22
*** xjuan has joined #buildstream16:26
jonathanmawmilloni: Do I understand correctly that you're using commit "98eb7381c8713157b549b6b18e07a3ffab987b69" from repo "git@gitlab.com:knownexus/buildstream.git", and when building the documentation for buildstream/sandbox/sandbox.py, you're getting an error message over the use of the label "sandboxing"16:27
jonathanmawis that correct?16:27
millonijonathanmaw, 98eb7381c8713157b549b6b18e07a3ffab987b69 from https://gitlab.com/knownexus/buildstream16:30
millonirunning16:30
millonimake16:30
millonifrom doc16:30
*** toscalix has quit IRC16:31
jonathanmawmilloni: hrm, I can't repeat your issue. Does it keep happening even if you do `git clean -dfx` from the "doc/" directory?16:38
juergbijennis: possibly better understandable function declarations are python_to_msg_byteorder(python_byteorder) and msg_to_python_byteorder(msg_byteorder)16:39
jennisYes jmac has suggested that16:43
jennisI will make the changes16:43
jennisThese functions are only used once each afaics16:44
jennisjuergbi, do you think this should be done in a separate commit or within the "dealt with redefined outername" commit?16:46
juergbia separate commit could make sense but it's fine with me either way16:47
gitlab-br-botbuildstream: merge request (214-filter-workspacing->master: WIP: Make workspace commands on a filter element transparently pass through to their build-dependency) #317 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/31716:49
millonijonathanmaw, yes16:50
jonathanmawmilloni: hrm, can you show me the output of `git cat-file -p 98eb7381c8713157b549b6b18e07a3ffab987b69^{tree}` please?16:54
jonathanmawcache key collision is unlikely, but I'd like to rule it out16:55
gitlab-br-botbuildstream: merge request (239-use-pylint-for-linting->master: Resolve "Use pylint for linting") #283 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/28317:10
jennisjuergbi, I've made all the changes you've suggested and a few more17:10
jennisOne thing to note, there was one redefined builtin I could not seem to fix, wonder if you could take a look?17:11
* jennis finds the right link17:11
jennishttps://gitlab.com/BuildStream/buildstream/blob/master/buildstream/utils.py#L82517:11
jennisjuergbi ^^ that's from the main repo, on my branch I've disabled the redefined builtin warning17:12
juergbiah, python standard library itself wouldn't pass that checker...17:14
juergbii don't think we can really do anything else about this17:15
jennisPlease could you expand?17:16
jennisBecause if it's to do with the decorator, the pylintrc file addresses contextmanager-decorators17:17
juergbitempfile.mkdtemp() accepts a `dir` keyword argument17:17
juergbithat's part of the python standard library, not defined by us17:17
juergbithat's assuming it's about dir. or do you see it contextmanager-related?17:17
jennishttps://gitlab.com/jennis/buildstream/blob/239-use-pylint-for-linting/.pylintrc#L16817:18
jennisahh no it is about dir17:18
jennisjust a coincidence that it's using that decorator as I remember seeing that in the pylintrc file17:19
*** dominic has quit IRC17:19
millonijonathanmaw, https://paste.codethink.co.uk/?418217:21
juergbijennis: we could possibly change our parameter but still use dir= for the standard library call. however, i think in this case it's better to be consistent with the python standard library and have the pylint disable17:22
jennisjuergbi: Ok, I agree17:23
jonathanmawmilloni: hrm, that matches what I had. What does `git status --ignored` say when called from the root of the repository?17:26
juergbijennis: gitlab is not responding, so here my last comment: You're now adding the `Lock` import in "pylint - dealt with import warnings" and then removing it in a separate commit. That doesn't make sense. Please remove it from the former commit, i.e., squash the two together.17:27
juergbi(the import doesn't exist in master anymore)17:28
jennismhmm I'll just reset to the penultimate commit17:28
jennisnot sure where it would have been added17:28
juergbiin an earlier version you moved the import to a different place. in master the import was removed. in your rebase to master, the reordering (removed line and added line) became an added import line17:30
jennisI see17:30
juergbiit's just a small rebase fallout17:30
jennisall done (:17:31
juergbii'm surprised that there is no unused import warning when running pylint on that earlier commit17:31
jennisjuergbi ^17:31
gitlab-br-botbuildstream: merge request (239-use-pylint-for-linting->master: Resolve "Use pylint for linting") #283 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/28317:31
jennisDid you just run pylint on that commit?17:32
juergbita17:32
juergbiyes17:32
jennismhm17:32
jennisthat is odd17:32
jennisaha17:33
jennisthere is still a section messages we would like to enable at some point in the pylintrc file17:33
jennisso unused import is actually disabled17:33
jonathanmawmilloni: I've got to go now. If you post your results I'll check them in the IRC logs when I get back.17:34
millonithanks jonathanmaw17:35
jennisjuergbi, should I go through these?17:35
juergbijennis: ah, didn't realize that17:35
juergbii'd say let's first merge what we have17:35
*** jonathanmaw has quit IRC17:35
jennisok17:35
juergbii'm trying to fetch from gitlab, but it's acting up :/17:35
jennisyes it's being v slow for me17:36
gitlab-br-botbuildstream: issue #239 ("Use pylint for linting") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/23917:54
gitlab-br-botbuildstream: merge request (239-use-pylint-for-linting->master: Resolve "Use pylint for linting") #283 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/28317:54
gitlab-br-botbuildstream: merge request (jmac/build-uid-2->master: WIP: Sandbox configuration key to specify uid/gid for sandbox) #318 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/31818:08
gitlab-br-botbuildstream: merge request (jmac/build-uid->master: WIP: Specify custom UID for build sandbox in elements) #301 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/30118:08
*** tiago has quit IRC18:28
*** toscalix has joined #buildstream19:30
*** toscalix has quit IRC19:34
*** xjuan has quit IRC20:25
*** mcatanzaro has quit IRC22:50
*** valentind has quit IRC22:59
*** aday has quit IRC23:39

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!