| *** mohan43u has quit IRC | 00:05 | |
| *** mohan43u has joined #buildstream | 00:09 | |
| *** tristan has quit IRC | 01:18 | |
| *** tristan has joined #buildstream | 01:18 | |
| *** tristan has quit IRC | 01:56 | |
| gitlab-br-bot | buildstream: merge request (jjardon/index_problems->master: Use toctree instead normal list so correct index always appear in the left menu) #345 changed state ("closed"): https://gitlab.com/BuildStream/buildstream/merge_requests/345 | 02:27 |
|---|---|---|
| *** tristan has joined #buildstream | 02:31 | |
| *** tristan has quit IRC | 03:14 | |
| *** tristan has joined #buildstream | 03:22 | |
| gitlab-br-bot | buildstream: issue #322 ("Pylint should probably complain about our ostree imports in CI") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/322 | 03:30 |
| gitlab-br-bot | buildstream: issue #314 ("buildstream should exit inmediatelly if the format is not supported") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/314 | 03:45 |
| *** benbrown has joined #buildstream | 06:42 | |
| *** valentind has joined #buildstream | 07:09 | |
| *** valentind has quit IRC | 07:34 | |
| gitlab-br-bot | buildstream: merge request (146-use-minimum-python-version-3-4-for-integration-tests->master: Resolve "Use minimum python version (3.4) for integration tests") #266 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/266 | 08:25 |
| gitlab-br-bot | buildstream: merge request (146-use-minimum-python-version-3-4-for-integration-tests->master: Resolve "Use minimum python version (3.4) for integration tests") #266 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/266 | 08:25 |
| tristan | Wasn't this fixed: https://gitlab.com/BuildStream/buildstream/issues/257 ? | 08:28 |
| tristan | seems still open, I think it was fixed, though | 08:28 |
| * tlater never noticed an MR for it | 08:30 | |
| jennis | tristan, hi, I'm getting a bit confused about how invoking.html is generated. You've filed the issue regarding the broken output of the man pages, which are generated with click_man. And this broken output is the same broken output that is seen in invoking.html | 08:30 |
| tristan | jennis, I put two links there to upstream bug reports, did I put the same link twice ? | 08:31 |
| jennis | However, looking at invoking.rst I assume that it generates it with sphinx_click directly from _frontend/cli.py, is that correct? | 08:31 |
| tristan | Yes, it's generated by the metadata encoded into python with the decorators in _frontend/cli.py | 08:31 |
| tristan | As is the --help output, automated by click | 08:32 |
| jennis | ok. So man pages with click_man, invoking.html with sphinx-click | 08:32 |
| tristan | tlater, I dont know if there was an MR for it, but I think it must be solved... lemme just check | 08:32 |
| jennis | and both have the same broken output | 08:32 |
| tristan | tlater, confirmed, it is *still* doing that; did we merge the version stuff ? | 08:33 |
| tlater | tristan: Version stuff is merged, but didn't touch that | 08:33 |
| tristan | that at least was corrected to not overwrite the file for no reason | 08:33 |
| tlater | Yep, it was :) | 08:34 |
| tristan | I would have thought it might be fixed as a consequence of that :-/ | 08:34 |
| tristan | oh well | 08:34 |
| * tristan was sure that, being as trivial as it is, it must have been fixed by now | 08:34 | |
| tlater | It being trivial is probably half the reason why it isn't fixed, just makes it feel like very low priority. | 08:36 |
| tlater | I think aiden was looking at it? Probably been caught up in other things though. | 08:36 |
| tristan | Some times, it's the accumulation of many minor annoyances that amount to a lot of frustration... | 08:37 |
| * tristan will try to make a run at fixing a lot of trivial things | 08:37 | |
| * tristan still also has a pending refactor of _pipeline.py and how it interacts with the frontend, regarding selections of elements... which desparately needs to be done | 08:38 | |
| tristan | business logic related to artifact caches seems to be bubbling up too far into the _pipeline.py module too | 08:38 |
| tristan | this makes it harder to review and understand the code (and consequently, to contribute to it) | 08:38 |
| tlater | tristan: On issue #322 - pylint only does static analysis. This means that if the types are purely generated on the fly, pylint should complain about the types not existing in the first place | 08:51 |
| tlater | In either case, that issue came up after we saw pylint complaining about importing arpy - this gi import follows the exact same pattern | 08:51 |
| tlater | jennis suspects it doesn't come up with a warning because we disable warnings for that class in general, but it's worth ensuring that pylint is doing what it should | 08:52 |
| *** Nexus has joined #buildstream | 08:54 | |
| *** Nexus has joined #buildstream | 08:55 | |
| *** jmac has joined #buildstream | 08:55 | |
| gitlab-br-bot | buildstream: issue #322 ("Pylint should probably complain about our ostree imports in CI") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/322 | 08:58 |
| tristan | tlater, okay, reopened. | 08:58 |
| tristan | tlater, you dont think perhaps pylint was smart enough to detect that this, though ? | 08:58 |
| tristan | I suppose it needs investigation | 08:59 |
| tlater | tristan: We've seen it dumb enough not to, so yeah, I'd like to investigate when I get to it | 08:59 |
| tristan | I suspect this is done by overriding how import works, similar to pluginsbase | 09:00 |
| tristan | So I wouldnt be surprised if the linter was smart enough to ignore such things | 09:01 |
| *** noisecell has joined #buildstream | 09:02 | |
| *** bethw has joined #buildstream | 09:05 | |
| tristan | tlater, I think this is quite satisfactory at this stage, wouldnt you ? https://gitlab.com/BuildStream/buildstream/issues/171 | 09:11 |
| tlater | tristan: It could be made a little bit faster still, I think, but yeah, it's quite good now | 09:12 |
| *** aday has joined #buildstream | 09:14 | |
| *** aday has quit IRC | 09:16 | |
| *** aday has joined #buildstream | 09:20 | |
| gitlab-br-bot | buildstream: merge request (jmac/performance-ci->master: WIP: Add benchmark to CI test and document CI.) #341 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/341 | 09:20 |
| tristan | tlater, good enough to close the bug I think ? | 09:21 |
| tlater | Yeah, I agree | 09:22 |
| tristan | I think part of the perceived slowness (on our part), is that you often have to tab twice in order to see a completion | 09:22 |
| *** aday has quit IRC | 09:22 | |
| tristan | if so, and if that can be rectified; we should consider that as a separate thing | 09:22 |
| * tristan closes that one | 09:22 | |
| gitlab-br-bot | buildstream: issue #171 ("bash completions are slow") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/171 | 09:23 |
| *** aday has joined #buildstream | 09:25 | |
| laurence | tlater, not urgent, but I was looking at this re workspaces - https://gitlab.com/BuildStream/buildstream/issues/209, and wondering which bits were complete, so i added check boxes. | 09:26 |
| laurence | hope that's okay | 09:26 |
| * tlater thought that issue was solved, in fact, so this neat :) | 09:27 | |
| tlater | Long shot, but does you have any thoughts on the errors I'm getting in this CI? https://gitlab.com/BuildStream/buildstream/-/jobs/59895250 | 09:29 |
| tlater | It seems like ruamel.yaml is incapable of serializing `set` objects in python3.4 | 09:29 |
| tlater | Workaround is just to use list, but there's a pretty sizable performance downgrade | 09:29 |
| * tlater would preferably do something else, but can't think of anything | 09:30 | |
| tristan | tlater, I think this is completely solved by your integration test refactor, right ? https://gitlab.com/BuildStream/buildstream/issues/126 | 09:32 |
| tlater | tristan: Yep, we only check for filenames anymore | 09:33 |
| *** aday has quit IRC | 09:33 | |
| tristan | and I think what we check depends a lot on the case, they are more focused on what they are testing | 09:34 |
| gitlab-br-bot | buildstream: issue #126 ("Relax integration tests a bit") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/126 | 09:34 |
| gitlab-br-bot | buildstream: merge request (soft-arpy-dependency->master: Soften arpy dependency) #342 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/342 | 09:39 |
| juergbi | tlater: is it possible that the issue is old ruamel in Debian, not Python 3.4? just a wild guess | 09:42 |
| tlater | juergbi: I'm not sure, thought we installed it through pip, but it's a good guess | 09:42 |
| * tlater takes a look at the image | 09:42 | |
| tlater | tristan: Would you mind reviewing an MR or two? They are quite small... | 09:42 |
| tlater | https://gitlab.com/BuildStream/buildstream/merge_requests/342 and https://gitlab.com/BuildStream/buildstream/merge_requests/319 | 09:42 |
| juergbi | it's installed with apt | 09:42 |
| tlater | Ah | 09:42 |
| tlater | Yeah, that would make sense | 09:43 |
| tlater | juergbi: That leaves the question, should we work around this or update our image to use an up-to-date ruamel? | 09:43 |
| juergbi | we anyway install some packages with pip, so I'd just move that to the pip installation | 09:44 |
| tlater | Nexus: Do you happen to know what your to-be-merged documentation suggests? Installing ruamel.py via apt or pip? | 09:44 |
| juergbi | yes, we need to make sure it matches the documentation | 09:44 |
| tristan | tlater, 342 done... | 09:45 |
| Nexus | tlater: apt | 09:45 |
| tlater | Egh, unfortunate | 09:46 |
| tlater | Nexus: Ta, that probably means we should work around it, or change the doc | 09:46 |
| juergbi | that can be changed as well. if we anyway include both apt and pip installs in the documentation, I don't see an issue moving a package between the two | 09:46 |
| Nexus | at this point, my docs are horribly out of date | 09:46 |
| Nexus | so much so that the MRs have been pulled | 09:47 |
| juergbi | if we currently don't need pip at all according to the documentation, then we may have to reconsider | 09:47 |
| tlater | Ok... | 09:47 |
| Nexus | (removed) | 09:47 |
| Nexus | probably shouldnt use git termanology | 09:47 |
| tlater | Nexus: Comment on what juergbi says? I suppose we could change the doc to use pip, but it would be nice to stick to apt if we can | 09:47 |
| *** aday has joined #buildstream | 09:48 | |
| Nexus | juergbi: i believe pip is only used to install bst | 09:48 |
| juergbi | right, the documentation recommends installing buildstream itself with pip, but that automatically pulls in dependencies | 09:49 |
| tristan | tlater, 319 done, I think we just go ahead and land 319 as is | 09:49 |
| juergbi | so can't we just remove ruamel from the apt line? | 09:49 |
| tlater | tristan: ta :) | 09:49 |
| * tlater will get to rewording 342 in a bit | 09:49 | |
| tristan | juergbi, I think there were some problems with pip installation of some of the libs on some platforms | 09:49 |
| tristan | hence why some things ended up recommended to install with package manager | 09:50 |
| juergbi | ah, I thought ruamel was pure Python | 09:50 |
| Nexus | tlater: i think we should arrange a sit down to discuss workspaces | 09:50 |
| juergbi | anyway, we should first test whether the ruamel version is even the issue. it was just a wild guess | 09:50 |
| tristan | juergbi, but I'm not sure it's a great argument, as one can still install via pip; it should be instead made to "just work" | 09:50 |
| * tlater thinks using distro package manager versions is better practice anyway | 09:50 | |
| tristan | juergbi, ruamel.yaml is pure python, probably we need to pin the version at something specific though, those install docs predate a lot of discovery | 09:51 |
| juergbi | if they are up-to-date, sure.. | 09:51 |
| tristan | I just recall that there "was problems" | 09:51 |
| tlater | tristan: Would you like it to "just work" for someone refusing to use pip in debian-8? | 09:51 |
| juergbi | the install documentation is also not for Debian 8, though, it's for Buster+, so ruamel in the distro might be just fine there | 09:52 |
| tristan | tlater, I dont know what our distribution story is for installing buildstream on a distro, yet | 09:52 |
| juergbi | it's just our 3.4 testing image that has this oldstable Debian | 09:52 |
| tristan | tlater, one thing is sure; we cannot reliably abide by the idea that "The version of a dependency which the distro 'foo' has chosen, is the one which will work for BuildStream" | 09:52 |
| tristan | tlater, I think currently there is techniques with venv for people who want to be particularly difficult, but I removed that from docs as it was not really used, and quite verbose | 09:53 |
| tlater | tristan: I was just wondering if we should support the versions of dependencies that a few supported distros ship | 09:54 |
| juergbi | have we written down what distros we actually (want to) support? this should be useful for future determinations when to drop Python 3.4 e.g., but also for other dependency versions | 09:54 |
| gitlab-br-bot | buildstream: merge request (jmac/performance-ci->master: WIP: Add benchmark to CI test and document CI.) #341 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/341 | 09:54 |
| tlater | We've stated that we'd support debian-8 a while ago, which is the entire reason for this MR | 09:54 |
| juergbi | ah, really? I thought Python 3.4 dependency was due to other (enterprise?) distros | 09:55 |
| tristan | tlater, I have a feeling that ultimately, our distro targetting packaging should include a pip-using step which downloads the exact things we intend to use, and that gets bundled up into a release tarball, which gets installed in such a way that we have carried the exact versions of things we intend to use | 09:55 |
| tristan | just a feeling, but it's hard to walk the line between pip and distros | 09:55 |
| tristan | we need to focus on reliability more than making either fan club happy, I think | 09:56 |
| juergbi | definitely | 09:56 |
| tristan | :) | 09:56 |
| tlater | Hm, shouldn't we stick to distro packages then, as those tend to be more reliable? | 09:56 |
| persia | If they are supported. | 09:57 |
| juergbi | depends on your definition of reliable.. | 09:57 |
| tristan | tlater, distro packages "a version of something", with a philosophy that "things" respect the rules of API compatibility, usually | 09:57 |
| juergbi | from the upstream perspective, distro packages are highly unreliable | 09:57 |
| juergbi | as they vary across the different distros and distro versions | 09:57 |
| persia | My suggestion is to assign folk to care for pypi, and each distro, working with the relvant teams to make sure there are consistent working things in those places. | 09:57 |
| tristan | tlater, python things tend to on the other hand, completely disregard this, and say "Use version == x.y.z" rather than "Use version >= x.y" | 09:58 |
| *** aday has quit IRC | 09:58 | |
| tlater | I suppose it makes sense to defer figuring out packaging to the actual distros, so yeah, pip makes sense | 09:59 |
| tristan | persia, that implies though, that BuildStream itself must be functional against a wider variety of versions of things, and we need to keep track of those externally decided versions | 09:59 |
| persia | tristan: I only meant to imply that if BuildStream cares whether it works in some environment, the best way to achieve this is to have someone take care of it who is associated with a BuildStream identity. | 10:00 |
| persia | Leaving things to folk not involved in the BuildStream identity is a great way to be surprised when something doesn't work because of unexpected versions. | 10:00 |
| tristan | persia, however, assuming that on the distro, it is always possible to parallel install multiple versions of the same python module... we could work with upstreams to ensure they have the versions BuildStream will use | 10:00 |
| persia | Many distros aren't like that. | 10:01 |
| tristan | Anyway, I dont think we'll solve this problem today, but it's something we should think about before 1.2 | 10:01 |
| *** aday has joined #buildstream | 10:02 | |
| persia | Also, I'm not sure what you mean by "upstream" there: for most of the distro-related interactions I have, distros are "downstream", with the rare exception of derivative distros that are incredlbly far downstream and so will sometimes use "upstream" to indicate an ancestor. | 10:03 |
| *** aday has joined #buildstream | 10:03 | |
| tristan | persia, misnomer/typo/thingy, oops :) | 10:06 |
| persia | Ah, OK :) | 10:06 |
| *** tristan has quit IRC | 10:43 | |
| tlater | juergbi: looks like the ruamel version actually wasn't the issue | 10:46 |
| tlater | :| | 10:46 |
| Nexus | is it possible to run all tests after x? so ./setup.py --addopts "after test-foo" | 10:46 |
| tlater | Nexus: Define "after" | 10:46 |
| Nexus | tlater: i know that the first 40% of tests pass and are unaffected by my work, so i don't wanna run them most of the time | 10:47 |
| Nexus | but i want to the the rest | 10:47 |
| tlater | Nexus: Not aware of any such ting, might be something in the pytest library though | 10:47 |
| Nexus | kk | 10:48 |
| tlater | If you'd like to test individual/a pattern of tests there's '-k' though, which I find more useful for this sort of scenario | 10:48 |
| Nexus | yeah i use that a lot, but it's limited to the location of the test, and that seems to have nothing to do with the execution order :/ | 10:48 |
| *** aday has quit IRC | 10:51 | |
| Nexus | ... hm | 10:51 |
| * Nexus seems t have a hanging test | 10:51 | |
| Nexus | tests/artifactcache/junctions.py::test_push_pull | 10:51 |
| gitlab-br-bot | buildstream: merge request (146-use-minimum-python-version-3-4-for-integration-tests->master: Resolve "Use minimum python version (3.4) for integration tests") #266 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/266 | 10:52 |
| tlater | Nexus: Gah, not again :| | 10:52 |
| Nexus | :) | 10:52 |
| gitlab-br-bot | buildstream: merge request (jmac/performance-ci->master: WIP: Add benchmark to CI test and document CI.) #341 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/341 | 10:54 |
| *** mohan43u has quit IRC | 11:05 | |
| gitlab-br-bot | buildstream: merge request (146-use-minimum-python-version-3-4-for-integration-tests->master: Resolve "Use minimum python version (3.4) for integration tests") #266 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/266 | 11:06 |
| *** mohan43u has joined #buildstream | 11:08 | |
| gitlab-br-bot | buildstream: merge request (jmac/performance-ci->master: Add benchmark to CI test and document CI.) #341 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/341 | 11:11 |
| gitlab-br-bot | buildstream: merge request (146-use-minimum-python-version-3-4-for-integration-tests->master: Resolve "Use minimum python version (3.4) for integration tests") #266 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/266 | 11:16 |
| tlater | juergbi: I think the python3.4 MR is finally ready https://gitlab.com/BuildStream/buildstream/merge_requests/266 | 11:28 |
| tlater | I went with encoding lists in the end, as it does appear to be a python3.4 bug | 11:28 |
| tlater | Or at least something not supported by ruamel atm | 11:28 |
| * tlater considers writing an issue, but for now the performance shouldn't be too bad | 11:29 | |
| tlater | It could just be better | 11:29 |
| tlater | Hrm, I need a feature from 2017.11 for my artifact cache expiry branch | 11:43 |
| tlater | Unless there is a better way to list local refs, would it be ok to bump the ostree version to ^? | 11:43 |
| *** aday has joined #buildstream | 11:54 | |
| *** aday has quit IRC | 11:55 | |
| *** aday has joined #buildstream | 11:55 | |
| jennis | Just a sanity check here: When the build log says "Caching Artifact " does this imply local? And when it says "Pushing Artifact" + "Preparing compressed archive" + "Sending Artifact", that's all remote cache? | 11:59 |
| tlater | jennis: Yep | 12:00 |
| *** aday has quit IRC | 12:05 | |
| gitlab-br-bot | buildstream: merge request (jennis/ensure_consistent_build_log_output->master: element.py: Make artifact lowercase in time_activity()) #346 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/346 | 12:09 |
| jennis | ^^ tlater do you mind reviewing this, just a very tiny bug bear that I wanted to quickly fixed | 12:12 |
| jennis | I've tested a build with that branch and it does as expected | 12:12 |
| tlater | jennis: Sure, give me a minute | 12:12 |
| jennis | No rush | 12:13 |
| * Nexus personally prefers the capital 'A' | 12:13 | |
| jennis | Nexus, it's not consistent with all other build output though | 12:13 |
| tlater | Yeah, most build output has all caps, sticking with that seems best | 12:13 |
| * Nexus is confused | 12:14 | |
| gitlab-br-bot | buildstream: issue #146 ("Use minimum python version (3.4) for integration tests") changed state ("closed") https://gitlab.com/BuildStream/buildstream/issues/146 | 12:14 |
| gitlab-br-bot | buildstream: merge request (146-use-minimum-python-version-3-4-for-integration-tests->master: Resolve "Use minimum python version (3.4) for integration tests") #266 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/266 | 12:14 |
| jennis | tlater, huh? | 12:14 |
| jennis | https://paste.baserock.org/alewahekal | 12:14 |
| tlater | :O ty juergbi! | 12:14 |
| tlater | jennis, Nexus: Sorry, that's what I meant | 12:15 |
| tlater | Stupid fingers being faster than my brain | 12:15 |
| jennis | hehe | 12:15 |
| Nexus | WHO ARE YOU AGREEING WITH!?!? | 12:15 |
| jennis | Nexus, look at this: https://paste.baserock.org/alewahekal | 12:15 |
| jennis | You'll see what I mean about the build output | 12:16 |
| jennis | It's always first word capitalised, the rest lowercase | 12:16 |
| tlater | Nexus: Agreeing with jennis | 12:16 |
| Nexus | ok, thats fine | 12:16 |
| Nexus | jennis: i know, i'm just saying i prefer capitals on all | 12:16 |
| tlater | jennis: On that note, no issues with the MR, but I can't merge it x) | 12:16 |
| Nexus | i.e. would change all to be Pascal Case | 12:17 |
| tlater | jennis: Did you check if any tests rely on that text being uppercase? | 12:31 |
| tlater | Looks like there's one that's failing | 12:31 |
| Nexus | that sounds like a horrific test | 12:33 |
| * tlater can't see why else a test would fail here, just guessing | 12:34 | |
| tlater | I restarted the pipeline expecting this not to be the case | 12:34 |
| Nexus | tests/frontend/push.py::test_push_after_pull | 12:40 |
| Nexus | failed in that CI | 12:40 |
| * Nexus wonders why jennis's changes broke the test | 12:43 | |
| Nexus | jennis: try rebasing? | 12:44 |
| *** aday has joined #buildstream | 12:49 | |
| *** mohan43u has quit IRC | 13:10 | |
| jennis | mhmm I didn't will look at that now tlater | 13:10 |
| *** mohan43u has joined #buildstream | 13:13 | |
| jennis | How long do hastebin links last? | 13:14 |
| jennis | 30 days ^^ | 13:14 |
| * tlater wonders why ostree lists a ref but then refuses to remove it :| | 13:15 | |
| gitlab-br-bot | buildstream: merge request (jennis/ensure_consistent_build_log_output->master: element.py: Make artifact lowercase in time_activity()) #346 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/346 | 13:18 |
| jennis | tlater, any thoughts on this | 13:23 |
| jennis | I can't see anything related to "artifact" or "Artifact" in that test | 13:24 |
| tlater | jennis: It's probably in the assert_pushed_elements helper | 13:26 |
| tlater | Which probably parses the buildstream output to see which elements were pushed | 13:27 |
| jennis | found it lol | 13:31 |
| jennis | tlater, ok it passes the push.py test now. I'll just make sure it passes all of the other tests then push | 13:41 |
| Nexus | js, it now being BST in england is messing up all of my email filters with buildstream... | 13:42 |
| tlater | I like how they named a time zone after us. | 13:43 |
| jennis | haha yeah i'm getting incorrect timestamps on my remote server | 13:43 |
| * Nexus politely requests we rename bst to bs | 13:46 | |
| * Nexus doesnt care which one | 13:46 | |
| jennis | utc? | 13:47 |
| Nexus | but noone uses it | 13:48 |
| gitlab-br-bot | buildstream: merge request (jennis/ensure_consistent_build_log_output->master: element.py: Make artifact lowercase in time_activity()) #346 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/346 | 13:50 |
| jennis | Could someone give that a quick review ^^, just case changes. | 13:52 |
| jennis | Please (: | 13:52 |
| jmac | Looking | 13:52 |
| jmac | Ok, looks good to me | 13:53 |
| *** noisecell has quit IRC | 13:59 | |
| *** Prince781 has joined #buildstream | 14:02 | |
| tlater | Could I enlist someone to take a look at an ostree problem with me? | 14:05 |
| tlater | I'm really struggling to remove a commit that I'm certain exists in the cache | 14:05 |
| tlater | Ostree happily lists it, but tells me it doesn't exist when I ask it to remove it :( | 14:06 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 14:06 |
| * tlater creates an MR to point to tests | 14:06 | |
| Nexus | tlater: i can help if you think i can | 14:12 |
| tlater | Feel free to, I'm at a loss: https://gitlab.com/BuildStream/buildstream/-/jobs/59975248 | 14:13 |
| tlater | Should take a bit to finish, but then you can see the exceptions... | 14:13 |
| jennis | awwwh tlater, its definitely not as simple as a rm -rf | 14:14 |
| jennis | https://paste.baserock.org/foxubajujo | 14:14 |
| jennis | It really doesn't like it... It kills my docker container | 14:14 |
| Nexus | "simaple_bst_project" | 14:14 |
| Nexus | typo maybe? | 14:15 |
| tlater | jennis: No idea what's wrong | 14:15 |
| tlater | *But* the docker container exiting probably means that the daemon crashes | 14:15 |
| tlater | So check your docker container output for debugging info | 14:15 |
| *** noisecell has joined #buildstream | 14:15 | |
| tlater | Docker also keeps handy logs for you if you need them | 14:16 |
| jennis | looooool | 14:16 |
| tlater | i.e. `docker log *` | 14:16 |
| Nexus | is the type really not an issue? | 14:16 |
| jennis | well spotted Nexus | 14:16 |
| Nexus | woo \o/ | 14:16 |
| * tlater finds it amusing that Nexus mistypes typo | 14:16 | |
| Nexus | SHUT UP | 14:16 |
| Nexus | NOONE SAW | 14:16 |
| jennis | hahaha | 14:16 |
| Nexus | why | 14:17 |
| Nexus | do | 14:17 |
| Nexus | the | 14:17 |
| Nexus | tests | 14:17 |
| Nexus | take | 14:17 |
| Nexus | so | 14:17 |
| Nexus | long‽ | 14:17 |
| jennis | linting + pytest + benchmarks | 14:19 |
| tlater | Hmm | 14:19 |
| * tlater discovered something interesting | 14:19 | |
| Nexus | is it 42? | 14:20 |
| Nexus | because i already knew that | 14:20 |
| * tlater might have been feeding this thing the wrong key this entire time | 14:21 | |
| Nexus | ... | 14:21 |
| tlater | Oh, nevermind, I haven't | 14:21 |
| Nexus | ... | 14:21 |
| Nexus | why is pylint complaining about my empty except? | 14:22 |
| Nexus | "bare-except" | 14:22 |
| tlater | Nexus: You could capture system exceptions with that | 14:22 |
| tlater | It's bad, don't do it | 14:23 |
| tlater | Unless you have a *very* good reason to | 14:23 |
| Nexus | i do have a *very* good reason to | 14:23 |
| tlater | You're too lazy to figure out the exact exception you'd like to catch? | 14:23 |
| Nexus | i'm planning on removing the line as soon as this suite passes and therefore am too lazy to figure out the exact exception you'd like to catch if i'm going to delete the work anyway | 14:24 |
| Nexus | s/you'd/i'd/ | 14:24 |
| tlater | ;p | 14:24 |
| Nexus | :) | 14:24 |
| jennis | uhhhhhh | 14:33 |
| jennis | I've managed a stop gap remotely | 14:34 |
| Nexus | gj? | 14:34 |
| jennis | on the remote* | 14:34 |
| jennis | It's just very specific... So not sure if it's a good idea | 14:34 |
| jennis | Would like to discuss | 14:34 |
| Nexus | then by all means do so | 14:35 |
| * jennis will type it up elsewhere then C&P | 14:35 | |
| jennis | 1 min | 14:35 |
| Nexus | kk | 14:35 |
| jennis | So basically the remote cache has the directory /data/artifacts | 14:40 |
| Nexus | woo ci failed | 14:40 |
| jennis | This contains: config extensions objects refs state summary tmp | 14:40 |
| jennis | Where config and summary are files, the rest are directories. | 14:40 |
| jennis | So I tried a `rm -rf /data/artifacts/*` then tried to build something that pushes to this. I got complains about there not being a config file. | 14:40 |
| jennis | Luckily, the only files in a "fresh" remote cache are config summary, so I decided to remove all files (EXCEPT config and summary) whilst maintaining the directory structure with: `find /data/artifacts/*/ ! -type d -exec rm '{}' \;` | 14:40 |
| jennis | Now, when I re-build, it pushes to a somewhat "fresh" remote cache again. | 14:40 |
| jennis | The only problem is, I'm not sure whether specific projects will require files in some of the other directories to remain undeleted...? | 14:40 |
| Nexus | can't you just put summary and config one dir up or put everything else in a subdir? | 14:42 |
| jennis | / does this seem like a good temporary solution | 14:42 |
| Nexus | that seems neater to me | 14:42 |
| jennis | What and use a simpler command? | 14:42 |
| jennis | No because I need to retain subdirectory structure | 14:42 |
| tlater | jennis: ostree refs --delete | 14:42 |
| jennis | because there are subdirs in refs/ that NEED to be there | 14:42 |
| tlater | ^ Or rather ostree --repo=/data/artifacts refs --delete | 14:43 |
| tlater | That should remove every ref in the repo | 14:43 |
| jennis | :| | 14:43 |
| tlater | Of course, if you can, access that via pygobject | 14:43 |
| tlater | And then you're already half done deleting individual artifacts :) | 14:43 |
| jennis | So don't use: `find /data/artifacts/*/ ! -type d -exec rm '{}' \;` | 14:43 |
| tlater | jennis: It's a cute hack, and probably works | 14:44 |
| tlater | But the above uses ostree, so it's likely safer | 14:44 |
| tlater | You might have to `ostree prune` after | 14:44 |
| jennis | <tlater> Of course, if you can, access that via pygobject | 14:44 |
| jennis | This is something you're unsure of, yes? | 14:44 |
| tlater | I know you can | 14:45 |
| tlater | But I'm not sure you'd like to go down that rabbit hole just yet, otherwise you wouldn't be trying to rm -rf, no? | 14:45 |
| tlater | jennis: Referenceis here: https://lazka.github.io/pgi-docs/#OSTree-1.0 | 14:45 |
| jennis | ok thanks tlater | 14:46 |
| Nexus | tlater: did you modify the yaml constructor? | 14:46 |
| tlater | Nexus: Which one, when? | 14:46 |
| jennis | I'm going to try and use the commands you suggested instead of my hack and see if that works | 14:46 |
| tlater | Ah, my CI finished | 14:47 |
| tlater | Nexus: I think that stuff is unrelated and due to the python3.4 switch T.T | 14:47 |
| * tlater tries to find the error he's looking at | 14:47 | |
| tlater | Well, nice, it doesn't show up in debian-8 | 14:48 |
| Nexus | yay? | 14:48 |
| * tlater will have to make it debian compatible first, then | 14:48 | |
| Nexus | yay more waiting | 14:48 |
| tlater | Nexus: Probably will do once I know what's causing this | 14:49 |
| Nexus | pastebin me the error? | 14:49 |
| tlater | Yeah, I can do that, hang | 14:49 |
| * Nexus hangs | 14:49 | |
| tlater | Nexus: https://hastebin.com/raw/iyofupobix | 14:54 |
| * tlater wonders where jennis dug up hastebin | 14:54 | |
| * tlater enjoys hastebin | 14:54 | |
| Nexus | ok | 14:55 |
| Nexus | so it's failing to delete the sha | 14:55 |
| Nexus | does '7075411a7610d309fe99ee03e7022765648be3dd7933f6c7585e6c014daefa8e' definitely exist? | 14:56 |
| tlater | Yep | 14:56 |
| tlater | At least, ostree lists it when I ask it what refs exist | 14:56 |
| Nexus | run it with -s? | 14:56 |
| * tlater doesn't really see the point, as the entire log is in the output already | 14:57 | |
| tlater | Even the entire test is in the output | 14:57 |
| jennis | tlater, I saw paulsherwood use it | 14:57 |
| Nexus | i was hoping for more info on what happened when it TRIED to delete the sah | 14:58 |
| Nexus | sha* | 14:58 |
| jennis | It's everything you want in a pastebin | 14:58 |
| tlater | Nexus: Well, the reason there's no massive error is because I catch it for extra context | 14:58 |
| Nexus | myy first thought ws permissions | 14:58 |
| tlater | I can remove that catch | 14:58 |
| Nexus | my second thought was that i can't type | 14:59 |
| tlater | jennis: It doesn't seem to do much beyond markdown syntax, but I've not read much into it | 14:59 |
| tlater | For the most part it seems really cool | 14:59 |
| tlater | Might start hosting one myself | 14:59 |
| tlater | Nexus: This is without my error-catching: https://hastebin.com/ayovotocan.md | 15:02 |
| Nexus | Deleting object 37c245160bf8187fad8196a3838566549056e7a40114ea807066a21fad4a6e70.commit: unlinkat(37/c245160bf8187fad8196a3838566549056e7a40114ea807066a21fad4a6e70.commit): No such file or directory (1) | 15:03 |
| tlater | Yep | 15:03 |
| Nexus | looks like OSTree.ObjectType.COMMIT isn't a dir where it should be? | 15:03 |
| tlater | Nexus: Note this isn't os.rmtree or anything like that | 15:04 |
| tlater | Calling OSTree.Repo.delete_object() | 15:04 |
| tlater | Which we give a ref that ostree itself gave us | 15:04 |
| juergbi | tlater: might it be a dangling ref? | 15:04 |
| juergbi | i.e., the commit is listed in refs but the commit object is not actually in the repo? | 15:05 |
| tlater | juergbi: The artifact for that commit is built in the invocation just before this one | 15:05 |
| tlater | How would I end up with a dangling ref in that case? | 15:05 |
| juergbi | that shouldn't be the case but have you made sure that's not the case? | 15:06 |
| tlater | Hrm, not sure how I would. I have quadruple checked that I'm getting the refs I'm expecting | 15:07 |
| juergbi | i.e., have you checked whether the repo is hosed for some reason before the delete or whether the delete itself is behaving incorrectly? | 15:07 |
| tlater | I could try to prune and then list the refs before I delete | 15:07 |
| juergbi | tlater: can you run ostree fsck before the delete? | 15:07 |
| tlater | Oh, that exists? | 15:07 |
| * tlater will try | 15:07 | |
| tlater | ta jeu | 15:07 |
| tlater | juergbi: | 15:08 |
| tlater | Also ta Nexus, of course :) | 15:09 |
| * Nexus didn't help at all | 15:09 | |
| * Nexus is still reading the OSTree.Repo.delete_object() docs | 15:10 | |
| Nexus | is something failing to mount? | 15:13 |
| * tlater doesn't think so, as the build succeeds | 15:14 | |
| tlater | juergbi: fsck fails, though I really don't understand why: https://hastebin.com/avitalusif.sql | 15:15 |
| Nexus | 16:13 < Nexus> is something failing to mount? | 15:15 |
| Nexus | unmounting itself afterwards? | 15:15 |
| juergbi | tlater: so the commit object is missing. why that is the case I can't say without more information | 15:15 |
| tlater | Hm, well, at least it's a lead I suppose | 15:16 |
| juergbi | is this happening in pushed code? | 15:16 |
| Nexus | check just before the build finished to see if the commit object is there, and then just after | 15:17 |
| Nexus | see when (if at all) it stops existing | 15:17 |
| juergbi | +1 | 15:17 |
| tlater | juergbi: Yep, though the CI fails for unrelated errors: https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 15:17 |
| * tlater doesn't think there will be anything obvious to see though, still very WIP | 15:18 | |
| juergbi | tlater: _ostree.remove removes the commit without removing the ref | 15:18 |
| juergbi | i.e., it makes the ref dangling | 15:19 |
| tlater | juergbi: I think ostree.prune gets rid of it after? | 15:19 |
| juergbi | that's the other way round | 15:19 |
| juergbi | if you delete the ref, prune will delete the commit object | 15:19 |
| tlater | Ah! | 15:19 |
| tlater | Nonetheless, the method fails the first time it's called | 15:19 |
| tlater | But I suppose let's see if I figure out how to use prune correctly | 15:20 |
| juergbi | btw: I don't expect prune to be very cheap, so if you're deleting multiple refs, you probably want to first delete all these refs and then call prune() once | 15:20 |
| juergbi | definitely sounds odd it would fail the first time | 15:21 |
| juergbi | does delete_object fail or prune? | 15:21 |
| tlater | delete_object does | 15:21 |
| tlater | Hmm... I wonder if all this time I was looking for `prune_static_deltas` | 15:22 |
| tlater | It doesn't even prune anything T.T | 15:25 |
| Nexus | :p gj | 15:29 |
| juergbi | we don't use static-deltas, afaik | 15:38 |
| juergbi | so there is nothing to prune | 15:38 |
| tlater | juergbi: I wasn't sure what static-deltas were, but clearly that's not useful | 15:40 |
| tlater | It looks like it wasn't the first iteration of removing though | 15:40 |
| juergbi | that would make sense | 15:40 |
| tlater | My safety margin wasn't large enough, so it started removing artifacts sooner than I thought it did | 15:40 |
| tlater | repo.set_ref_immediate(None, ref, None) solves everything \o/ | 15:41 |
| tlater | tyvm for the assistance juergbi :) | 15:41 |
| tlater | Also Nexus ;) | 15:41 |
| juergbi | yw | 15:41 |
| gitlab-br-bot | buildstream: merge request (135-expire-artifacts-in-local-cache->master: WIP: Resolve "Expire artifacts in local cache") #347 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/347 | 15:49 |
| *** bethw has quit IRC | 15:59 | |
| *** Prince781 has quit IRC | 16:00 | |
| *** Prince781 has joined #buildstream | 16:02 | |
| *** Prince781 has quit IRC | 16:11 | |
| *** bethw has joined #buildstream | 16:12 | |
| *** aday has quit IRC | 16:15 | |
| gitlab-br-bot | buildstream: merge request (jennis/ensure_consistent_build_log_output->master: element.py: Make artifact lowercase in time_activity()) #346 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/346 | 16:29 |
| *** mohan43u has quit IRC | 16:31 | |
| jennis | juergbi, if you have a second !346 should be mergeable. No worries if not :) | 16:31 |
| jennis | ^^ just a bug-bear fix | 16:31 |
| *** mohan43u has joined #buildstream | 16:32 | |
| juergbi | jennis: fyi, in general the goal is for tests to pass for every commit. we're not completely strict about this but e.g. here we should squash the two commits | 16:36 |
| jennis | Oh I see, I will do that now | 16:40 |
| jennis | ta juergbi | 16:40 |
| *** aday has joined #buildstream | 16:41 | |
| gitlab-br-bot | buildstream: merge request (jennis/ensure_consistent_build_log_output->master: element.py: Make artifact lowercase in time_activity()) #346 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/346 | 16:43 |
| jennis | there you go juergbi :) | 16:43 |
| *** mohan43u has quit IRC | 16:47 | |
| *** mohan43u has joined #buildstream | 16:48 | |
| jennis | tlater, still there? | 16:54 |
| tlater | Kindq | 16:55 |
| tlater | *kinda | 16:55 |
| jennis | hehe | 16:56 |
| tlater | Well, probably not anymore in 5 mins | 16:56 |
| jennis | So ostree refs give me my list of refs | 16:56 |
| jennis | now ostree refs --delete <something_from_that_list> works | 16:56 |
| gitlab-br-bot | buildstream: merge request (jennis/ensure_consistent_build_log_output->master: element.py: Make artifact lowercase in time_activity()) #346 changed state ("merged"): https://gitlab.com/BuildStream/buildstream/merge_requests/346 | 16:57 |
| jennis | Any ideas on how to remove all of them? | 16:57 |
| jennis | A simple * doesn't do the trick | 16:57 |
| jennis | and I'm failing to find examples online | 16:57 |
| tlater | I think giving *no* argument should do it | 16:57 |
| tlater | `ostree refs --delete` | 16:57 |
| tlater | Otherwise, sorry, not sure :( | 16:57 |
| jennis | doesn't work either :/ | 16:57 |
| tlater | The gobject interface certainly can do it | 16:58 |
| jennis | [root@c053b21a625b artifacts]# ostree refs --delete | 16:58 |
| jennis | error: At least one PREFIX is required when deleting refs | 16:58 |
| tlater | Aww | 16:58 |
| tlater | Might have to loop manually | 16:58 |
| jennis | Yeah that's an option I guess | 17:01 |
| jmac | I never found a good way of doing this from the command line | 17:02 |
| *** bethw has quit IRC | 17:04 | |
| *** tristan has joined #buildstream | 17:29 | |
| *** brlogger has joined #buildstream | 18:36 | |
| *** xjuan has joined #buildstream | 18:48 | |
| *** mohan43u has quit IRC | 19:05 | |
| *** mohan43u has joined #buildstream | 19:09 | |
| *** tristan has quit IRC | 19:14 | |
| *** Prince781 has joined #buildstream | 20:38 | |
| *** xjuan has quit IRC | 21:50 | |
| *** mohan43u has quit IRC | 22:07 | |
| *** mohan43u has joined #buildstream | 22:10 | |
| *** aday has quit IRC | 22:26 | |
| *** Prince781 has quit IRC | 23:01 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!