IRC logs for #baserock for Thursday, 2017-07-13

*** gtristan has quit IRC03:44
*** gtristan has joined #baserock04:14
*** noisecell has joined #baserock06:23
*** paulwaters_ has joined #baserock07:32
*** bruce has joined #baserock07:39
*** bruce is now known as Guest6556707:40
*** toscalix has joined #baserock07:51
*** rdale has joined #baserock08:23
*** jonathanmaw has joined #baserock08:30
*** jude_ has joined #baserock08:37
*** Guest65567 has quit IRC08:49
*** ctbruce has joined #baserock08:55
*** ssam2 has joined #baserock09:14
*** ChanServ sets mode: +v ssam209:14
*** CTtpollard has joined #baserock09:28
*** locallycompact has joined #baserock09:55
benbrown_ironfoot: wai u no amend commit10:00
ironfootbenbrown_: gah! lack of time!10:00
ironfootwill do so in my next spare moment10:01
ironfootbenbrown_: thanks for reviewing!10:01
*** locallycompact has quit IRC10:02
*** locallycompact has joined #baserock10:02
benbrown_ironfoot: Any idea if gbo is under a particularly high load?10:36
* benbrown_ is wondering why this pipeline is taking so long
ssam2~ # uptime10:36
ssam2 10:36:52 up 4 days, 10:09,  2 users,  load average: 4.83, 4.20, 3.9010:36
ssam2not too bad10:37
ironfootbenbrown_: looking at the log, looks like the mirroring of repos is going fast10:38
ironfootmaybe the host running the CI is slow?10:38
ironfootis being slow* (busy, etc)10:39
benbrown_ironfoot: yup, turns out they were using the gitlab shared runners, which are apparently super slooow11:36
ssam2anyone here want push access to the repo for testing?11:47
ssam2pm me your SSH private key if so11:47
ssam2don't do that11:48
ssam2but send me your public key11:48
*** noisecell has quit IRC11:48
* paulsher1ood will try once there's a web interface11:48
paulsher1ood(nb... downstream users sometimes want to download a specific artifact)11:48
paulsher1ood(independent of the build tool.... hence kbas has wildcard search and offers links to click on)11:50
ssam2i agree that it's needed, but have no scope in my current project to achieve it11:50
paulsher1oodis there no way just to reuse the kbas bottle stuff?11:52
* paulsher1ood still doesn't understand ostree in this context11:52
ssam2it might be a useful starting point11:52
ssam2but the files are stored in a different way, they're not just tar files on disk11:52
ssam2so providing a download link is more tricky11:53
paulsher1ooddoesn't the build tool already download artifacts anyway?11:53
ssam2yes, but using code from ostree11:54
ssam2the same reason that you can't just put a .git directory on a web server and have people download all the files from a specific commit11:54
ssam2cgit makes that possible, and I imagine it does so by doing `git archive` internally11:54
paulsher1oodso... is the artifact cache one huge .git directory, then?11:55
paulsher1ood(for all artifacts)11:55
ssam2in some ways, yes11:55
ssam2the implementation is different, but the idea is similar11:55
gtristanssam2, I think we can implement `bst pull` for pulling an artifact, and `bst push` for pushing artifacts which may have been built while offline (if that's desirable), in which case we could provide an interface through bst for downloading and checking out a specific artifact (if that's helpful)11:55
paulsher1oodgtristan: it's something, but there are situations where $user just needs artifact, has no bst11:56
gtristanpaulsher1ood, indeed; similar to git, objects that are identical have the same checksum; so an artifact cache can handle a lot more before ever needing to be pruned or wiped (this becomes more true the more reproducible your builds are, too)11:56
paulsher1oodyes... that's the upside in choosing ostree style storage, downside is dealing with these pesky use-cases :)11:57
gtristanpaulsher1ood, for a user who wants to download artifacts as tarballs from a web server; it's conceivable to have some function to extract and tar it up on the server11:57
ssam2`ostree export` exists, we'd just need to hook it up to a web form11:58
gtristanssam2, right, ever does it as tar iirc (so you keep your permissions and xattrs and such, and can do as regular user)11:58
gtristans/ever/even, even11:59
gtristanin any case; we can have such a fancy web-app in the gitlab BuildStream group, I only hope to not have the tool/ecosystem depending on that (i.e. it's just added sugar for interfacing with a webpage)12:01
ssam2yes... it could perhaps even be part of ostree12:01
ssam2although that would likely be much more work as the scope would grow and grow12:01
ssam2but we could keep that in mind12:01
gtristanssam2, I'm a bit torn about putting too much work there too, ostree is a primary use case but we're going to have to figure out (very soon) how to work with platforms it just wont run on12:02
ssam2agreed. i'm not committing any time right now :-) (although if it becomes a hard blocker for Baserock adopting BuildStream, there might be time left at the end of my project to do something)12:03
paulsher1oodcan you not just allow kbas as an option? :)12:03
* paulsher1ood gets his coat12:03
ssam2a tarball artifact cache is pretty much the plan for platforms where OSTree doesn't run though, so maybe12:04
gtristanssam2, I mean, if we're going to have a web-app, we'd like to keep control of that too, because we'll want it to support non-ostree cases, or; we're not sure what happens when it comes to that12:04
ssam2maybe... depends if multiple backends within a single artifact cache server is going to be useful for anyone12:04
gtristanssam2, what would be interesting though, is have artifact cache shares just run on linux (with ostree), even if buildstream needs to download/upload as tar on some exotic platforms12:05
ssam2i was just thinking that. if it can import and export tarballs then we're grand12:05
ssam2although then BuildStream on those platforms depends on said webapp12:05
gtristanright, we lose some performance on upload/download on platforms without ostree (cause we cant just upload/download objects that are needed, we need to send a whooole tarball)12:06
gtristanbut it's still something12:06
ssam2a single cache server that has two caches in incompatible formats seems a bit pointless. I guess if you had a mix of Linux and non-Linux builds, the other option would be just have two different cache servers... an ostree one and a tar/kbas one12:06
gtristanssam2, indeed, this is going to need some thought; we're going look at a PoC but I expect the first PoC to avoid concerns of artifact sharing12:07
ssam2if it's different platforms then there's little advantage to sharing anyway12:07
ssam2i.e. a Solaris machine has little use for prebuilt Linux artifacts12:07
gtristanssam2, also a good point12:07
jjardon seems trove (or something) is timeout every ~1m:10s12:13
jjardonfor every single repo12:13
jjardonIs there trove very busy atm?12:14
ssam2what tells you that it's timing out ?12:14
ssam2i don't see any errors logged12:14
ssam2and no errors in the journal on so far12:15
benbrown_"no tree from tree-server for 50893291621658f355bc5b4d450a8d06a563053d"12:15
benbrown_isn't that part of morph-cache-server?12:15
ssam2It used to be, certainly12:16
ssam2nothing in the journal at all12:17
benbrown_Has that recently been disabled? Or is it still going?12:17
ssam2seems to be running12:17
ssam2does the tree exist ?12:17
benbrown_ybd wouldn't be able to generate the cache-key if it didn't12:18
ssam2logging doesn't seem to be enabled for that service12:18
ssam2i've tried restrating it12:18
ssam2do you either of you want access to the server to debug further ?12:19
jjardonssam2: notthing tell me that, but its a bit strange there is always a gap of ~1m:10s between cloning repos12:23
ssam2possibly a 60 second timeout in the the client side code, and the server isn't responding at all ?12:24
ssam2try calling the cache server AP manually with curl and see if you get a response12:24
*** deep-book-gk_ has joined #baserock13:34
*** deep-book-gk_ has left #baserock13:34
benbrown_ssam2: I'd appreciate access, if that's alright? Not got too much time to look into it right now though.13:52
ssam2ok, if you send me your ssh key I can set it up13:53
ssam2to the ostree cache you mean or ?13:53
benbrown_gbo :)13:53
benbrown_ostree cache will become useful to me later though, I'm sure, but I'll request access to that at a later point :)13:54
ironfootbenbrown_: did you find anything?15:07
*** gtristan has quit IRC15:10
benbrown_ironfoot: I've yet to take a look15:16
ironfootgbo is robust ;)15:16
benbrown_busybox less makes me sad15:17
benbrown_hitting end just seems to kill it15:19
ironfootif you are looking at the journal is because it's taking time15:19
benbrown_nah, morph-cache-server logs15:19
ironfootthey might be long too15:19
benbrown_very long :)15:22
*** gtristan has joined #baserock15:26
benbrown_"curl: (7) Failed to connect to port 8080: Connection timed out"15:42
ssam2oh........ I know the issue15:44
ssam2sorry, all my fault15:44
ironfootit's going through the haproxy?15:44
ssam2I redid the security groups last night when doing the OStree artifact15:44
ssam2and thought "port 8080 isn't needed any more because we're not running morph-cache-server"15:44
ssam2forgetting that it serves trees as well as artifact15:44
ssam2ok, will fix15:44
ironfootssam2: note that changes in infrastructure.git should go through gitlab15:45
ironfoot(if any)15:45
ssam2 :-)15:45
ironfoottsk ironfoot, you didn't look15:46
ssam2benbrown_, try now15:49
benbrown_curl works, idk the API though, so I just get a server error :Þ15:50
ssam2sounds like an improvement :-)15:50
benbrown_indeed :)15:51
benbrown_ssam2: Ty15:51
gitlab-br-botybd: issue #262 ("ybd should be able to generate -debug rpms") changed state ("opened")
gitlab-br-botinfrastructure: merge request (sam/ostree->master: Add initial definitions) #5 changed state ("opened"):
gitlab-br-botinfrastructure: merge request (sam/ostree->master: Add definitions) #5 changed state ("opened"):
gitlab-br-botybd: issue #262 ("ybd should be able to generate -debug rpms") changed state ("closed")
*** ctbruce has quit IRC16:16
*** toscalix has quit IRC16:34
*** jonathanmaw has quit IRC17:40
*** locallycompact has quit IRC17:56
gitlab-br-botinfrastructure: merge request (sam/ostree->master: Add definitions) #5 changed state ("opened"):
*** ssam2 has quit IRC18:03
gitlab-br-botinfrastructure: merge request (sam/ostree->master: Add definitions) #5 changed state ("merged"):
*** locallycompact has joined #baserock18:41
gitlab-br-botlorries: merge request (pedro/lorry-controller-lorry->master: Add lorry-controller.git from Gitlab) #20 changed state ("opened"):
gitlab-br-botlorries: merge request (pedro/lorry-controller-lorry->master: Add lorry-controller.git from Gitlab) #20 changed state ("opened"):
gitlab-br-botlorries: merge request (pedro/lorry-controller-lorry->master: Add lorry-controller.git from Gitlab) #20 changed state ("merged"):
gitlab-br-botlorries: merge request (pedro/lorry-lorry->master: Add lorry.git from Gitlab) #21 changed state ("opened"):
ironfootbenbrown_: thanks for the reviews!19:44
ironfootyou have merged our first lorry in YAML :)19:44
*** locallycompact has quit IRC20:23
*** rdale has quit IRC23:23
*** gtristan has quit IRC23:37

Generated by 2.15.3 by Marius Gedminas - find it at!