| *** Prince781 has quit IRC | 00:13 | |
| *** Prince781 has joined #buildstream | 00:19 | |
| *** tristan has quit IRC | 01:28 | |
| *** Prince781 has quit IRC | 01:37 | |
| albfan[m] | Do we have buildstream account in twitter already? | 06:05 |
|---|---|---|
| albfan[m] | should I include https://gitlab.com/albfan/buildstream-flatpak-gtksourceview in doc examples. it is another flatpak deps + git project | 06:07 |
| *** bochecha_ has joined #buildstream | 06:46 | |
| *** coldtom has joined #buildstream | 07:51 | |
| *** Phil has joined #buildstream | 08:17 | |
| *** bethw has joined #buildstream | 08:27 | |
| gitlab-br-bot | buildstream: merge request (jmac/cas_virtual_directory->jmac/googlecas_and_virtual_directories_2: WIP: CAS-backed virtual directory implementation) #481 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/481 | 08:29 |
| *** abderrahim[m] has quit IRC | 08:39 | |
| *** krichter[m] has quit IRC | 08:39 | |
| *** albfan[m] has quit IRC | 08:39 | |
| *** m_22[m] has quit IRC | 08:39 | |
| *** theawless[m] has quit IRC | 08:39 | |
| *** asingh_[m] has quit IRC | 08:39 | |
| *** rafaelff[m] has quit IRC | 08:39 | |
| *** ernestask[m] has quit IRC | 08:39 | |
| *** alatiera has quit IRC | 08:39 | |
| *** inigomartinez has quit IRC | 08:39 | |
| *** kailueke[m] has quit IRC | 08:39 | |
| *** mattiasb has quit IRC | 08:39 | |
| *** ptomato[m] has quit IRC | 08:39 | |
| *** jjardon[m] has quit IRC | 08:39 | |
| *** waltervargas[m] has quit IRC | 08:39 | |
| *** pro[m] has quit IRC | 08:39 | |
| *** cgmcintyre[m] has quit IRC | 08:39 | |
| *** oknf[m] has quit IRC | 08:39 | |
| *** segfault3[m] has quit IRC | 08:39 | |
| *** theawless[m] has joined #buildstream | 08:49 | |
| *** bochecha_ has quit IRC | 08:54 | |
| *** jonathanmaw has joined #buildstream | 09:08 | |
| *** aday has joined #buildstream | 09:14 | |
| *** aday has quit IRC | 09:15 | |
| *** dominic has joined #buildstream | 09:16 | |
| *** aday has joined #buildstream | 09:26 | |
| *** Phil has quit IRC | 09:42 | |
| *** Phil has joined #buildstream | 09:43 | |
| *** jonathanmaw has quit IRC | 09:44 | |
| *** jonathanmaw has joined #buildstream | 10:00 | |
| *** Phil has quit IRC | 10:22 | |
| *** Phil has joined #buildstream | 10:23 | |
| *** bochecha has joined #buildstream | 10:30 | |
| *** rafaelff[m] has joined #buildstream | 12:40 | |
| *** kailueke[m] has joined #buildstream | 12:40 | |
| *** alatiera has joined #buildstream | 12:40 | |
| *** connorshea[m] has joined #buildstream | 12:40 | |
| *** mattiasb has joined #buildstream | 12:40 | |
| *** abderrahim[m] has joined #buildstream | 12:40 | |
| *** asingh_[m] has joined #buildstream | 12:40 | |
| *** ssssam[m] has joined #buildstream | 12:40 | |
| *** albfan[m] has joined #buildstream | 12:40 | |
| *** waltervargas[m] has joined #buildstream | 12:40 | |
| *** oknf[m] has joined #buildstream | 12:40 | |
| *** Guest29107 has joined #buildstream | 12:40 | |
| *** ptomato[m] has joined #buildstream | 12:40 | |
| *** cgmcintyre[m] has joined #buildstream | 12:40 | |
| *** inigomartinez has joined #buildstream | 12:40 | |
| *** jjardon[m] has joined #buildstream | 12:40 | |
| *** segfault3[m] has joined #buildstream | 12:40 | |
| *** m_22[m] has joined #buildstream | 12:40 | |
| *** Demos[m] has joined #buildstream | 12:40 | |
| *** aday has quit IRC | 13:08 | |
| jonathanmaw | Hi valentind, I've been working on implementing source mirroring with API changes instead of weird hacks, and I'm mostly happy with it now (branch at https://gitlab.com/BuildStream/buildstream/tree/jonathan/mirror-client-sourcedownloader) | 13:49 |
| jonathanmaw | though I'm having a thorny issue with versioning | 13:49 |
| jonathanmaw | AIUI, buildstream's versioning logic is based around checking whether buildstream is too old for a given plugin, rather than whether a plugin is too old for buildstream | 13:50 |
| laurence | reminder: we have the monthly buildstream irc team meeting in 10 minutes time, over in #buildstream-meetings | 13:50 |
| jonathanmaw | my specific issue is calling the likes of Source.fetch(), where I've added an alias_override arg | 13:51 |
| laurence | (don't forget the s on the end of meetings :)) | 13:51 |
| jonathanmaw | But I'm struggling to figure out the correct way to check whether a source plugin is too old to support the new function signature | 13:51 |
| tlater | Hm, I wonder if we could do a reverse lookup on `bst workspace close` if the user attempts to close a directory | 13:53 |
| tlater | Personally, I keep trying to close the directory because conceptually that's the workspace | 13:53 |
| jmac | I've written a small tool to inspect & extract hashes from CAS which might be useful to people working on it: https://gitlab.com/jmacarthur/cas-inspector | 13:54 |
| jmac | (Might post to the list later too) | 13:54 |
| jonathanmaw | valentind: at worst, I suppose I can catch a TypeError for the sources not supporting that function signature, but I'll keep looking for a nicer way | 13:57 |
| jonathanmaw | :/ Plugin.BST_FORMAT_VERSION is almost what I'm looking for, but that is for whether the config field has a new format, not an API change in the plugin | 13:58 |
| jonathanmaw | hrm, I suppose if I add Source.BST_API_VERSION = 0, and then set all the source plugins' versions to 1, I can determine that any source with BST_API_VERSION of 0 hasn't been changed | 14:03 |
| cs_shadow | gitlab is throwing 500 again :( | 14:16 |
| paulsherwood | they're moving to gcloud | 14:16 |
| cs_shadow | like right now? | 14:16 |
| paulsherwood | unclear | 14:16 |
| paulsherwood | https://about.gitlab.com/2018/06/25/moving-to-gcp/ | 14:17 |
| * paulsherwood asked (perhaps unfairly) if this is related to the MS github deal | 14:17 | |
| cs_shadow | thanks for the link. | 14:19 |
| cs_shadow | It says "The failover is currently scheduled for Saturday, July 28, 2018." so I guess this is just _normal_ GitLab | 14:19 |
| cs_shadow | it's mildly amusing how about.gitlab.com is perfectly fine but status.gitlab.com is throwing 502 | 14:19 |
| paulsherwood | cs_shadow: to be fair they've seen a significant uptick in usage since the ms announcement, which seems to have caught them somewhat | 14:20 |
| cs_shadow | that's fair. Perhaps this is all part of an evil plan by GitHub to drown GitLab in their own success ;) | 14:22 |
| paulsherwood | lol | 14:23 |
| jonathanmaw | juergbi: Can you give input on my versioning conundrum? | 14:40 |
| valentind | laurence, I am sorry I missed the meeting. I had something I did not expect. | 14:41 |
| valentind | laurence, Is the transcript posted somewhere? | 14:42 |
| laurence | valentind, dw, i think that we will have enough discussion f2f next week at GUADEC | 14:45 |
| valentind | jonathanmaw, I will have a look at the branch later today. | 14:46 |
| jonathanmaw | thanks valentind | 14:47 |
| *** Demos[m] has quit IRC | 14:55 | |
| *** rafaelff[m] has quit IRC | 14:55 | |
| *** mattiasb has quit IRC | 14:55 | |
| *** m_22[m] has quit IRC | 14:55 | |
| *** connorshea[m] has quit IRC | 14:55 | |
| *** ptomato[m] has quit IRC | 14:55 | |
| *** albfan[m] has quit IRC | 14:55 | |
| *** theawless[m] has quit IRC | 14:55 | |
| *** waltervargas[m] has quit IRC | 14:55 | |
| *** abderrahim[m] has quit IRC | 14:55 | |
| *** inigomartinez has quit IRC | 14:55 | |
| *** kailueke[m] has quit IRC | 14:55 | |
| *** Guest29107 has quit IRC | 14:55 | |
| *** alatiera has quit IRC | 14:55 | |
| *** oknf[m] has quit IRC | 14:55 | |
| *** ssssam[m] has quit IRC | 14:55 | |
| *** jjardon[m] has quit IRC | 14:55 | |
| *** segfault3[m] has quit IRC | 14:55 | |
| *** asingh_[m] has quit IRC | 14:55 | |
| *** cgmcintyre[m] has quit IRC | 14:55 | |
| *** coldtom has quit IRC | 14:58 | |
| *** awacheux[m] has joined #buildstream | 15:04 | |
| *** coldtom has joined #buildstream | 15:06 | |
| *** tristan has joined #buildstream | 15:12 | |
| *** coldtom has quit IRC | 15:17 | |
| *** coldtom has joined #buildstream | 15:17 | |
| valentind | tristan, To be able to detect if a "source" in from an include in a junction, do you think it is fine to keep track of the original project in the Provenance? Or there is smarter than that. | 15:28 |
| tristan | valentind, I think that is a novel idea | 15:45 |
| valentind | I am not sure it all make sense. | 15:46 |
| valentind | But we need it. | 15:46 |
| tristan | Well, the provenance is supposed to tell you where each final resolved bit of loaded YAML originally came from | 15:46 |
| tristan | and it's used both for serializing back at track time (without project.refs in play), and for error messages | 15:47 |
| tristan | I mean, it seems a decent place | 15:47 |
| tristan | *however* | 15:47 |
| tristan | valentind, I have to admit that it sounds quite memory intensive now that I think of it | 15:47 |
| tristan | it would be nicer if we had a shared object for provenance objects to hold the filename and project portions | 15:48 |
| valentind | OK. I was going to propose that. | 15:49 |
| valentind | I will try that. | 15:49 |
| tristan | I have to say that, I suspect there still remains some fringe bugs where provenance is not properly resolved on a node | 15:49 |
| tristan | however this has never been known to happen with a source | 15:49 |
| * tristan thinks that _yaml.composite_node() is imperfect when it comes to crawling lists | 15:50 | |
| laurence | seems that my emails aren't getting through to the buildstream mailing list | 16:10 |
| laurence | not sure where exactly the issue is, but i'm not having any other email issues atm | 16:11 |
| jmac | Can someone remind me where juerg's CAS Fuse work is? | 16:12 |
| valentind | tristan, it seems tracking errors are translated into warnings. Should we keep this behavior for inline tracking with fragment from junctions? | 16:18 |
| jmac | ^ found it | 16:20 |
| valentind | https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/_scheduler/trackqueue.py#L65 | 16:20 |
| *** Prince781 has joined #buildstream | 16:26 | |
| *** coldtom has quit IRC | 16:37 | |
| tristan | valentind, right, the system errors we can get from writing the file happen at an inopportune place | 16:37 |
| *** bethw has quit IRC | 16:37 | |
| tristan | valentind, they need to become an error, but possibly from a high priority timeout | 16:38 |
| tristan | iirc | 16:38 |
| tristan | valentind, yes I think that falls under the same category | 16:38 |
| tristan | valentind, and we explicitly forbid includes in project.refs / junction.refs to keep things simpler | 16:39 |
| valentind | tristan, should we save the last exception found and raise it after all results have been treated? | 16:42 |
| tristan | You mean to turn that into an error ? | 16:44 |
| valentind | tristan, I have to read a bit more the code of the scheduler. I get confused. | 16:44 |
| tristan | valentind, that code happens as a side effect of a process completing, and with fresh eyes it looks like that comment predates some more recent changes | 16:46 |
| tristan | at this point, raising an exception there is safe, but still not altogether the most desirable | 16:46 |
| valentind | We do catch BstError higher. | 16:46 |
| tristan | it will be treated as a post processing failure | 16:46 |
| valentind | in queue.py | 16:46 |
| tristan | right, we started doing that | 16:46 |
| tristan | that's the one that handles core side responses to completions of tasks, exceptions there used to result in hangs hehe | 16:47 |
| tristan | So yeah it's decent but it could still be better | 16:47 |
| tristan | those failures have no recourse for retry, for instance | 16:48 |
| tristan | valentind, I suppose that a post processing failure is still a better behavior than the warning ? | 16:49 |
| tristan | Not sure | 16:49 |
| tristan | there be some dragons around there too | 16:49 |
| *** Phil has quit IRC | 16:51 | |
| tristan | i.e. fail vs. fail permanently distinction is missing (sources re-downloaded when tarball in fact has a mismatching checksum), the errors being collected as messages at the frontend level, not synchronized with task completion | 16:51 |
| * tristan should probably fix that problem | 16:51 | |
| *** jonathanmaw has quit IRC | 17:10 | |
| *** paulsherwood has quit IRC | 17:26 | |
| *** jmac has quit IRC | 17:26 | |
| *** Nexus has quit IRC | 17:26 | |
| *** Nexus has joined #buildstream | 17:26 | |
| *** adds68 has quit IRC | 17:27 | |
| *** paulsherwood has joined #buildstream | 17:27 | |
| *** jmac has joined #buildstream | 17:27 | |
| *** dominic has quit IRC | 17:28 | |
| *** finn has quit IRC | 17:29 | |
| *** tlater has quit IRC | 18:03 | |
| *** milloni has quit IRC | 18:03 | |
| *** aiden has quit IRC | 18:03 | |
| *** laurence has quit IRC | 18:03 | |
| *** valentind has quit IRC | 18:03 | |
| *** aiden has joined #buildstream | 18:04 | |
| *** aiden has quit IRC | 18:05 | |
| *** aiden has joined #buildstream | 18:05 | |
| *** laurence has joined #buildstream | 18:06 | |
| *** paulsherwood has quit IRC | 18:07 | |
| *** jmac has quit IRC | 18:07 | |
| *** Nexus has quit IRC | 18:07 | |
| *** benbrown has quit IRC | 18:07 | |
| *** milloni has joined #buildstream | 18:08 | |
| *** tlater has joined #buildstream | 18:08 | |
| *** valentind has joined #buildstream | 18:09 | |
| *** Prince781 has quit IRC | 18:16 | |
| *** Guest45103 has joined #buildstream | 18:33 | |
| *** ptomato[m] has joined #buildstream | 18:33 | |
| *** theawless[m] has joined #buildstream | 18:33 | |
| *** waltervargas[m] has joined #buildstream | 18:33 | |
| *** m_22[m] has joined #buildstream | 18:33 | |
| *** rafaelff[m] has joined #buildstream | 18:33 | |
| *** abderrahim[m] has joined #buildstream | 18:33 | |
| *** asingh_[m] has joined #buildstream | 18:33 | |
| *** connorshea[m] has joined #buildstream | 18:33 | |
| *** mattiasb has joined #buildstream | 18:33 | |
| *** kailueke[m] has joined #buildstream | 18:33 | |
| *** cgmcintyre[m] has joined #buildstream | 18:33 | |
| *** inigomartinez has joined #buildstream | 18:33 | |
| *** jjardon[m] has joined #buildstream | 18:33 | |
| *** alatiera has joined #buildstream | 18:33 | |
| *** segfault3[m] has joined #buildstream | 18:33 | |
| *** ssssam[m] has joined #buildstream | 18:33 | |
| *** albfan[m] has joined #buildstream | 18:34 | |
| *** Demos[m] has joined #buildstream | 18:34 | |
| *** oknf[m] has joined #buildstream | 18:34 | |
| *** jcampbell has quit IRC | 19:20 | |
| *** jcampbell has joined #buildstream | 19:22 | |
| *** tristan has quit IRC | 20:27 | |
| *** paulsherwood has joined #buildstream | 21:03 | |
| *** bochecha has quit IRC | 21:33 | |
| *** bochecha has joined #buildstream | 21:34 | |
| *** bochecha has joined #buildstream | 21:34 | |
| *** bochecha has joined #buildstream | 21:35 | |
| *** bochecha has quit IRC | 21:36 | |
| *** bochecha has joined #buildstream | 21:36 | |
| *** bochecha has joined #buildstream | 21:37 | |
| *** bochecha has quit IRC | 21:37 | |
| *** bochecha has joined #buildstream | 21:37 | |
| *** bochecha has quit IRC | 21:41 | |
| *** tristan has joined #buildstream | 22:24 | |
| tlater | tristan: Btw, for these scheduler resources, would you use classes or just a nested dict? It seems like one of those cases where a class is totally overkill, but I seem to gravitate to that a lot... | 23:16 |
| tlater | I suppose it's easy enough to change that in the future, so I'm wasting time even considering that | 23:17 |
| tristan | woops | 23:31 |
| tristan | oh not that long ago :) | 23:31 |
| tristan | tlater, I don't understand about nested dicts and scheduler resources | 23:32 |
| tristan | tlater, I remember the "resources" API we discussed yesterday | 23:32 |
| tristan | tlater, you need a resource to describe itself and have a list of named resources, and you're talking about the data structure to declare those per resource behaviors ? | 23:33 |
| tristan | i.e. since we said that "some resources are limited" | 23:33 |
| tristan | for the purpose of --fetchers / --builders / --pushers | 23:33 |
| tlater | Yes, basically | 23:35 |
| tlater | What I've gone with is to let the scheduler maintain a dict of these, and to use tokens on all resources. I'm currently thinking about how to work with exclusive access, I'll probably allow acquiring a special token for each resource, and the scheduler will have to make sure that we make room for that process | 23:37 |
| tlater | (Sorry if this is too deep into implementation, always hard to step back and try to let someone have a peek in my mind (: ) | 23:39 |
| tlater | The workflow is; when someone wants to start a job, they first ask the scheduler for a set of "resources" the job may require. | 23:41 |
| tlater | The scheduler then goes and checks its little list of how many tokens for each resource it has lying around | 23:42 |
| tristan | Hmmm | 23:42 |
| tristan | tlater, it's not just "A job exposes what resources it wants, and whether it wants them exclusively" ? | 23:42 |
| tristan | tlater, and let the scheduler work out what to do ? | 23:42 |
| tristan | tlater, or, does it mean the scheduler needs to be selective in which jobs it pops off the queues ? | 23:43 |
| tlater | It has to be a bit selective so that we can handle the priority case still | 23:43 |
| tristan | i.e. so that the jobs stay on queue until we really process it ? | 23:43 |
| tristan | right | 23:43 |
| tristan | tlater, a place for that could be _scheduler/__init__.py or a new file | 23:44 |
| tristan | tlater, if we can achieve something where the queues no longer ever import the scheduler for any reason, that would be good | 23:44 |
| tristan | i.e. not a challenge for you, but a generally good direction for the scheduler | 23:45 |
| tlater | Hm, I suppose the functionality is nicely encapsulated, so that will work | 23:45 |
| tristan | I think after my last refactor there still was reasons... the job executes methods on the scheduler directly for instance | 23:46 |
| tristan | that should ideally be different and layered, jobs could just have their own callbacks for the scheduler to handle itself | 23:46 |
| tlater | I suspect that this will want another refactor when events come in | 23:47 |
| tristan | but anyway, I don't expect a revolution, just saying ideally we want to walk towards a world where jobs dont import queues or scheduler, and queue also doesnt import scheduler | 23:47 |
| tlater | I noticed that the scheduler/queues were suddenly a lot less reliant on the frontend, too... Or I suppose the other way around | 23:48 |
| tlater | I guess there has been some refactoring there too since I started this work. | 23:49 |
| *** tristan has quit IRC | 23:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!