IRC logs for #buildstream for Thursday, 2020-06-25

*** tristan has quit IRC07:13
*** tristan has joined #buildstream07:18
*** ChanServ sets mode: +o tristan07:18
*** benschubert has joined #buildstream07:34
benschubertOk, for a threaded scheduler model, I am at 8 failures and 11 errors on the whole test suite. It's getting there :D08:02
tristanbenschubert, Sounds good !08:05
tristanI guess we can just go ahead with this then, but then there will be some benchmarking we should do...08:06
benschuberttristan: what I am not sure about is whether we want a "clean" MR, that cleans up some things more or if we want the bare minimum in and then improve?08:06
benschubertI'll benchmark before merging anyways :)08:06
tristanI have to admit that a nasty thought crossed my mind the other day08:06
benschubert?08:06
tristanI was thinking that... this thread-scheduling looks like a lot of work, which is a blocker for native windows support, but not *really* a blocker for 2.008:07
tristanI know, pesky brain08:07
tristanIt looks like it will be a great improvement anyway, I'll tell my brain to shut up :)08:08
benschuberttristan: that's true, and I know the aim is bst 2.0, but I also would really like a solid, well tested release, and these QOL improvements will probably help with that :D08:09
tristanYes, you don't have to tell me, believe me my heart is with you on that :)08:10
tristanI just got a call from the dark side of the force that's all :)08:10
* tristan wants to start really getting closer to 2.0, see it materialize on the horizon08:11
benschubertit happens :)08:11
benschubertI still don't have a solution for the plugins where I'm 99% satisfied BTW, I keep moving things around08:11
* benschubert really should site for a week and get that done08:11
tristanI am thinking a september release is now completely out of the question, but if we can get feature/major change completeness by some time in september... then a 2.0 in 2020 could be a reality08:12
tristanYeah the plugins is a big step08:13
benschubertI am doubtful around september, though that would indeed be good08:13
tristanI mean, december is usually mostly a bust, if you don't get your stable release out by the first days of december, it's next year08:14
benschubertbut if we miss the september/october milestone (Fedora 33/Ubuntu 20.10) I believe the next "important" milestone would be April/May on the opensource schedule right?08:14
benschubert(and I don't think september/october is feasible if we want a stable release)08:14
tristanI'm talking about feature / "major change" completeness for around september, hopefully by the end of september08:15
tristanIf that happens, I could see us having IRC team meetings again, and doing roll call on blockers08:15
tristanThere are a couple of months of ironing out kinks and being truly ready, after feature completeness and before release08:16
benschubertah yeah, ok seems good08:16
tristanOf course, this is mostly a wish statement at this point...08:17
tristanif we focus I think we can make it happen08:17
benschubertjuergbi: Where you involved in the vstorage_dir tests ? https://gitlab.com/BuildStream/buildstream/-/jobs/610202063 When run in randomized order there are a few failures and I am not sure what's the real cause. Would you have an idea?08:17
benschuberttristan: true08:18
juergbiI wasn't involved in writing the original test, although I think I have tweaked or extended it since then08:18
juergbidon't know otoh, can maybe take a look later08:19
benschubertsure, thanks08:19
tristanFWIW, the way I would see this fitting into the upstream world schedules is that, after stable is released (near close of 2020), this lets upstream projects start moving their master branches to use BuildStream 2, which would mean stable apps, runtimes, demo bootable images etc, around next summer, all using bst 208:19
juergbia propos bst 2.0 blockers, I'm making progress on the Remote Asset API front. initial push and pull of artifacts seems to work08:20
juergbi(but still a lot to do)08:20
benschubertjuergbi: that's awesome, I'm looking forward to my first build without any git checkout :D08:20
tristanAwesome08:20
benschuberttristan: true, there is more than "just" having bst2 in distributions08:20
juergbibenschubert: well, full integration into the sources with the corresponding plugin API extensions/changes is a whole other step08:21
tristanheh08:21
juergbifirst priority is moving from buildstream-specific protocol to remote asset API without changing the architecture08:21
juergbi(much)08:21
benschubertah gotcha08:22
tristanSo... the controversial topic, what about pulling the trigger on nuking the ability of plugins to write directly into the sandbox ?08:22
benschuberttristan: I still need to write an answer to the ML thread, but still need to finish reading it and look at all the plugins08:22
juergbiI have to read some more on that thread08:22
benschubertI should have an answer today/tomorrow08:23
tristanAnd, should we also remove a couple of APIs like... Element.set_public_data() and Element.get_variable() ?08:23
tristanbenschubert, I recommend this specific post: https://lists.apache.org/thread.html/r5af5f67b392ca5036f16f7ee2f33018f032d2615a8d5f814357d62df%40%3Cdev.buildstream.apache.org%3E08:23
tristanbenschubert, I took a long time yesterday trying to compose that, I think it's got a clear compelling argument08:24
tristanThe rest of the thread, read it but, it may be confusing to follow08:24
benschuberttristan: I'll read everything through08:24
benschubertone question that I haven't seen answer while skimming through though: How would you see the 'tar' element implemented with the change?08:24
tristanI don't believe there is any such thing as a 'tar' element08:25
tristanIt could be there is one, but then it should just be a script element really, which stages a build of `tar`08:25
benschubertwhich means now you need to have 'tar' available in your sandbox? What about docker, now you need buildah/tar again? (Just trying to get your whole view there :) )08:26
tristanYes - there should absolutely never be any data manipulation outside of the sandbox08:27
tristanDocker (or the 'oci' element), is one of the biggest offenders of this rule08:27
tristanIt's build is entirely non-deterministic, and as a result, it's output is hopelessly non-reproducible08:28
benschubertthe docker element is deterministic, if you run it twice your tar will even have the exact same sha08:28
benschubert*image not tar08:28
tristanIs that different from the oci element ?08:29
benschubertI have not look at the code of the oci element08:29
tristanAnd what about if you run it multiple times, each time with different hosts and different versions of python ?08:29
tristanWhere is docker08:29
tristanDoes it at least stage docker in a sandbox ?08:29
tristanI believe it should08:29
benschuberthttps://gitlab.com/BuildStream/bst-plugins-container/-/blob/master/src/bst_plugins_container/elements/docker_image.py08:29
benschubertthis is what it does08:29
benschubertwhich would not be possible under the new changes :)08:30
tristanbenschubert, definitely, that needs a rewrite08:30
tristanThat has to happen all inside the sandbox08:31
tristanIt is a terrible mistake that that was allowed to happen in the first place08:31
tristanHost python tarfile.open( ... ) manual python host manipulation, putting files in the sandbox08:31
tristanDefeats the purpose of using BuildStream in the first place08:32
tristanIf you wanted that, you could use `bst artifact checkout --tar > file.tar` in a script with host docker or such08:32
tristanthe oci plugin does similar08:33
benschubertI would agree if it was running arbitrary code, but given what is doing it's very similar to a "source" plugin in what it bring? (anywyas I'll answer on the ML once I'm all caught up)08:33
tristanIt is running arbitrary code, it's running tarfile.open() on host python08:33
tristanAlso, even if it happened to run code which was guaranteed by all upstreams involved to produce deterministic results, regardless of host of version of python/tool etc... the base principle of BuildStream is to assume that such guarantees cannot be made08:35
tristanThe tooling used to produce output is considered a part of the input08:35
tristans/of host of version of python/of host or version of python/08:36
*** santi has joined #buildstream08:46
tristanvalentind, the node passed to Variables() is never composited after being given to the Variables() constructor right ?09:43
tristanOr is it mutable ?09:43
tristanIf we were to change the constructor to copy the passed node, everything would still work ?09:43
valentindYes it would still work.09:44
tristanOk, yeah some grepping shows we only ever use it once in Element09:45
tristanSo it's just that in the case of junctions, things do not *have* to be fully resolved, only the things which the junction asks for has to be resolved09:46
* tristan is not sure why Variables implements __getitem__09:48
tristanmaybe it's a part of the current recursion09:49
tristanI think there are some wrong comments in this variables code :-S09:55
tristanhttps://gitlab.com/BuildStream/buildstream/-/blob/master/src/buildstream/_variables.pyx#L24309:56
tristanWhy on earth would it be useful to preserve "foo" and "baz" from that example string ??09:57
tristanwhoa, this appears terribly broken :-S10:02
tristanbenschubert, can you tell me what the API documenting comment should be for _expand_expstr(): https://gitlab.com/BuildStream/buildstream/-/blob/master/src/buildstream/_variables.pyx#L300 ?10:03
tristanit seems to have a copy/pasted doc comment from _expand_var()10:04
tristanand it's unclear what it's purpose is10:04
tristanI think I get it, but... it's quite wrong10:05
tristanOk... so basically, this takes the full list of values with any %{...} delimiters stripped, and treats each item as if it were a variable name, attempting to do an unconditional substitution on all text, and then joins them together10:06
tristanGotcha10:06
tristanbenschubert, The current code will expand "prefix%{prefix}" as "/usr/usr" instead of correctly expanding it as "prefix/usr"10:07
tristanAt least, that's my prediction from a reading of it10:07
benschuberttristan: let me have a look10:12
benschubertNo10:12
benschubertThe way it works here, is the list value ["x", "y", "z"] has this formt https://gitlab.com/BuildStream/buildstream/-/blob/master/src/buildstream/_variables.pyx#L243, which means that every even (0, 2, ...) indices are normal values, that should be taken directly, and every odd indices get expanded10:14
benschubertYou can see line 310 the handling for even indices and line 314 the handling for odd ones10:14
benschubertthat probably needs better documenting10:15
benschuberthttps://gitlab.com/BuildStream/buildstream/-/commit/8ae5e115a4693ff33248278f5702614ff6f3abee#135c57f9ad9406691ef175d76fc8ddcb09c4238d_289_319 was the previous comment which was more correct but not better10:17
benschubertI actually missed that comment misplace during the CR10:17
benschubertDoes that make sense?10:18
abderrahim[m]benschubert: since you're working on the scheduler, it would be nice if you could take a look at https://gitlab.com/BuildStream/buildstream/-/merge_requests/1970 (especially to avoid conflicts)10:18
abderrahim[m]juergbi: what are you using as a server for remote-asset? I didn't manage to find a server implementation10:19
* abderrahim[m] also wants to weigh in on the virtual directory thread but didn't get around to reading it10:20
juergbiabderrahim[m]: I added prototype code to buildbox-casd for remote asset support and then added a proxy for that part to bst-artifact-server as well10:20
benschubertabderrahim[m]: oh nice! I'll add this and review today thanks!10:21
abderrahim[m]juergbi: ah, please ping me when you have a public branch10:21
juergbiwill do10:22
coldtomi believe https://github.com/buchgr/bazel-remote has an experimental server implementation of the remote asset api too fwiw10:22
abderrahim[m]coldtom: it only implements Fetch and only fetches single files AFAICT10:23
juergbiyes, we also need push10:23
tristanHmmm, even and odd indices ??10:23
tristanLemme take a look at this closer10:24
benschuberttristan: in the 'value' list10:24
benschubertif you look at the while loop, we iterate through elements 2 by two10:24
benschubertbecause we handle each one differently10:24
tristanwhoa10:24
benschubertprobably needs better comments10:25
tristanSPLITTER.split("%{foo}%{bar}baz") -> ['', 'foo', '', 'bar', 'baz']10:25
tristanIndeed, talk about some black magic voodoo hiding out in there10:25
benschubertyep...10:25
tristanSo only even indices represent matches, is that it ??10:25
benschubertI'm not the one who came up with that, but it actually is around the fastest we can go10:25
benschuberteven are exact values that we take, odd are variables we need to expand10:26
tristancrazy10:26
benschubertand efficient :/10:26
benschubertIf you have an idea for a more maintainable and as fast approach I'm all for it though :)10:27
tristanI was thinking of a datatype to store the results instead of a list, but that only adds overhead10:27
tristane.g. a list of strings with a list of indices which are variables10:28
benschuberttristan: we're in full cython, we could do a struct or something, or special "classes" for variables and have a __str__ defined on them10:28
benschubertmight not be as fast, but could be pretty close10:28
tristanI don't think we need __str__ (that might just make things look more "magic" rather than explicit like plain C code), but having a type to represent this list could help10:29
benschubertyep10:30
tristancdef class ExpandedString():  ... cdef list words; cdef list variable_indices;10:31
tristanI wonder if that would add much overhead10:31
benschuberttristan: the best way to know is to experiment :D10:32
benschubertthanks to valentind we have a good project for testing variables now10:32
benschubertshould we actually have it somewhere as a test in BST? to ensure that some of it is correct and htus be able to also use it for performance10:32
tristanWe could keep the knowledge of this even/odd nonsense in the same function that does the re.split()10:32
benschubertand remove the 'empty' entries10:33
tristanWe should have it in the benchmarks repo, and finally finish the work in the benchmarks repo10:33
benschubertyeah10:33
tristanWe still havent integrated any CI of it10:34
benschubertthere is a lot of different benchmarking efforts that went and did things without pyshing towards a common thing10:34
tristanBut I don't want to have performance measurements in the test suite10:34
benschubertah I did not mean that :)10:34
tristanYeah anything that tests correctness belongs in the test suite10:34
benschubertI wonder if we should reset and start the benchmarking from scratch or something10:34
tristanBut I don't know what is missing in that regard10:34
tristanI've considered that for a time10:35
benschubertand then we do: words.copy(); for index in variable_indices: replace(workds[index]), return "".join(words)10:35
tristanBut I've been doing other work10:35
tristanI really want to use the logger, nanosecdond precision and log parsing for benchmarks10:35
benschubertI'd say let's postpone the benchmarking until we get the scheduler right, the plugins test sorted and the few other "big" chunks of work and come back to it later?10:35
tristando discrete measurements of targetted operations10:36
benschubertIs that really what we want? I'm more interested in E2e timings10:36
tristanI want to know about how much it costs to do a given routine by data set10:36
tristanLike if we have many elements, is our performance linear ?10:37
benschuberttrue10:37
benschubertwe'd need to decide on some datasets too10:37
tristanTo do that, I want to chop off irrelevant timings from the calculation10:37
tristanSo I would not count the time it takes to say, import pkg_resources10:37
tristanI want to know specifically how much time it takes to check for circular references in the load cycle, not counting the load cycle10:37
tristanFor large and small datasets, and compare10:38
tristanThis is dead easy to do10:38
benschubertthis doesn't need to use the `bst` command line though10:38
benschubertwe can actually do it inside10:38
tristanI've been given like 3 different resources over the years, had sessions in which I explained what needed to be done to achieve this, and every time, the whole thing just falls flat on it's face and said resource ends up working on something else10:38
tristanbenschubert, I think it would be easier (and more correct) to use `bst` though10:39
tristanAlso I think that leveraging the logger (with specially configured log output, as our userconfig allows), would yield true results10:39
tristani.e. we reduce the observer effect as much as possible10:40
tristanby not adding additional instrumentation codepaths10:40
tristanor reducing them10:40
tristanSince we already have the logger in play unconditionally, we should use that10:40
benschuberttristan: there are multiple things at stake here:10:40
benschubert- Timings of specific operations, that is for micro benchmarking, you'll likely want to repeat the operation many times and such. This is usually better done inside the code directly and not the cmdline10:40
benschubert- Timings of e2e, this is macro benchmarking and you get different information from each10:40
benschubertNah, using the logger is the worse thing possible for benchmarks10:40
benschubertI have timing more than 5 times longer using the logger vs dumping everything to /dev/null for macro benchmarking of buildstream10:41
tristanWe should be able to work with that10:41
tristanbenschubert, We could keep the log in RAM, we could allow configuration of the logger to avoid logging anything we don't care about for the purpose of the benchmark, etc10:42
benschubertwhich is easier done when importing the code directly, so you don't have to change too much of BuildStream user code for this10:42
tristanIf we had stable API for this, we could observe differences over the years10:43
tristanAlways show the load time/cyclic dependency check time, etc... for small/medium/large datasets, for 2.0, 2.2, 2.4, 2.610:43
benschubertThat's much trickier10:44
tristanAnd render pretty graphs10:44
benschubertthat's irrelevant10:44
benschubertyou don't want to keep the same kernel version, hardware, python version for the benchmarking over time10:44
tristanWell, I want to always see a graph of the last stable 2 releases + master10:44
tristanAt all times, without having to run the benchmarks myself10:44
tristanThe "current report"10:44
benschubertwhy would I be interested in python 3.6 timings when I'm using python3.9?10:44
tristanCrap, python10:44
benschubertpython3.8 is >20% faster than python3.6 on my benchmarks10:45
tristanfor the latest 2 stable versions, we could use the same version of python at least ?10:45
benschubertubuntu 20.04 with python3.8 is >10% faster than ubuntu 19.10 with python3.8 for my benchmarks10:45
tristanOf course you'd want dedicated hardware for the runner which does the benchmarking10:45
benschubertYeah10:45
tristanBut yeah, you definitely want to compare versions, unfortunately fast moving python makes this window small10:46
benschubertwhat I'm saying is, there are different levels of benchmarks that are interesting. for 'trends' over time, macro benchmarks through `bst` are easier and better and more stable, for debugging and trying to optimize, micro benchmarks are better10:46
tristanI would say that debugging is profiling10:47
benschubertbut what do you profile? :)10:47
tristanbenchmarking is categorizing operations and observing trends10:47
benschubertI can tell you that if you profile the variable expansion, it will seem a big part of the work, whereas in reality, it's not10:47
tristanWhen I see that for instance, resolving cyclic references is becomming exponential rather than linear, then I'll do profiling10:47
tristanin order to fix it and address the benchmarks10:48
tristanI definitely want the benchmarks to tell me *which part* of BuildStream is slow10:48
tristanNot just "Oh damn, this `bst build` command got slower"10:48
benschubert> I definitely want the benchmarks to tell me *which part* of BuildStream is slow10:49
benschubertYes, but you probably don't want that for the last 4 years. If you want to optimize, you want the current version and maybe 1-2 other points in the recent past10:49
tristanIf I see that a certain part of BuildStream is becoming slow, I can then see that it's a problem and I can profile that part10:49
tristanbenschubert, Right, that's what I mean latest 2 stable releases10:49
tristanOn say, a 6 month schedule10:49
tristanBut I don't want just "bst build"10:49
tristanI want a good handful of measurements in addition to "bst build"10:49
benschubertagain I don't disagree with that10:50
tristanAnyway, nobody did it, and I don't think we have time :(10:50
benschubertwhat I'm saying is that if you wnat ot benchmark the whole loading pre-scheduler, you'll be much better off having targeted benchmarks than things through `bst build`10:50
tristanHammering it in so that it sticks is probably a month of work for a single developer10:51
benschubertand then affects each run10:51
tristanMaybe less for a very simple implementation which has about 10% of the information I want to see10:51
benschuberttristan: my worry is that if we have this benchmarking information + profiling information already in place, things will get slower for _no gain to the user_10:52
benschubertand thus we should keep this separate10:52
tristanbenschubert, That's why I think we should have conditional logging statements and make that first class10:52
benschubertwell, that's again a runtime cost10:52
tristanWe should be able to tell the logger to not print anything except for "these specific start/stop messages"10:52
benschubertthe profiling check we have in place do have a measurable (though small) impact on the speed of buildstream10:53
tristanA single branch statement for a few hot codepaths should not be that expensive10:53
benschubertdo we _really_ want to add more of those?10:53
benschuberttristan: let's park this for now? In the end if we _do_ get an efficient, easy to maintain benchmarking setup in that I'd be fine. I will be picky on not affecting users though :)10:55
tristanbenschubert, Honestly, I've been wanting a more fancy `--debug` option too for a very long time and never got around to it, it would probably be similarly expensive10:56
tristane.g. it would be nice to say `--debug "core-topic1:core-topic2:element(compose):source(git)"`10:57
benschuberttristan: I like this idea and think it is worth it, as it's also good for users :)10:57
benschubertmight need to cythonize part of that if we want10:57
tristanAnd only have Plugin.debug() messages from the compose element, git plugin, and core debugging statements issued when debug("core-topic1", message...) was called10:57
tristanYou would have a one time parse at the beginning, reducing it to bits or booleans10:58
tristanwith a mapping, of course you would not use string compares at runtime10:58
tristanBut anyway, this would be as costly as adding a few conditionals for benchmarking (and could even be implemented with the same framework)10:59
benschuberttristan: that could be also easily cythonized to have it _real fast_ if we have performance problem afterwards10:59
tristanBut yes, agree about parking it - because nobody is going to do the work :'(10:59
benschubertand again: In the end if we _do_ get an efficient, easy to maintain benchmarking setup in that I'd be fine. I will be picky on not affecting users though :D10:59
tristanWeird, so cython is not very strict, you can define the types of variables you intend to use in a scope, but... you don't have to ?11:11
benschuberttristan: correct. Defining them means they'll be C variables when possible (strings, int, etc), otherwise it will default to PyObject, which is the least efficient11:15
benschubertSo for str, int, float, bool, list, dict, set, please always define11:16
tristanok11:21
tristanbenschubert, I was curious about https://gitlab.com/BuildStream/buildstream/-/blob/master/src/buildstream/_variables.pyx#L24511:21
tristanWhy 'splits' is not cdef list11:21
tristanBut I guess it's just an oversight11:21
benschubertcorrect, and we can therefore remove the <list> 5 lines later11:28
benschubertif we type it11:29
tristanGotcha11:42
tristanI think I'm getting somewhere, but I'll be interrupted by dinner11:42
tristanI think I can buy some performance also by using interns properly11:42
tristanThis code appears to try to use interns to save memory, but they should be used to speed up the dictionary lookups11:42
tristanHmmm, actually fairly good around that area, but I can see some gaps11:49
tristananyway, on a roll with variables finally... should have a good patch tomorrow11:50
*** tristan has quit IRC11:53
*** tristan has joined #buildstream12:06
*** ChanServ sets mode: +o tristan12:06
adds68hi, what is the default path used by te bst-artifact server?14:43
coldtomadds68: i think there's no default, it's just a cli arg https://docs.buildstream.build/master/using_configuring_cache_server.html#command-reference14:47
adds68oh thanks coldtom :)14:51
*** toscalix has joined #buildstream15:33
*** toscalix has quit IRC16:52
*** santi has quit IRC17:28
*** benschubert has quit IRC23:49

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!