| *** tristan has joined #buildstream | 05:15 | |
| *** ChanServ sets mode: +o tristan | 05:15 | |
| gitlab-br-bot | tristanvb approved MR !2026 (tristan/lazy-provenance->master: Refactor: Lazily instantiate ProvenanceInformation objects) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/2026 | 05:51 |
|---|---|---|
| tristan | Grrrr, I think overrides don't accept links for addressing subprojects | 07:00 |
| *** benschubert has joined #buildstream | 07:39 | |
| *** santi has joined #buildstream | 08:41 | |
| *** tristan has quit IRC | 08:52 | |
| *** hergertme has quit IRC | 09:05 | |
| *** hergertme has joined #buildstream | 09:13 | |
| *** tristan has joined #buildstream | 09:23 | |
| *** ChanServ sets mode: +o tristan | 09:23 | |
| *** SotK has quit IRC | 09:30 | |
| *** SotK has joined #buildstream | 09:30 | |
| *** phildawson has quit IRC | 11:43 | |
| tristan | benschubert, I don't understand this linting error: https://gitlab.com/BuildStream/buildstream/-/jobs/762755737 | 12:07 |
| tristan | "Class 'link_target' has no 'as_str' member (no-member)" | 12:07 |
| tristan | It's not a mypy, it's pylint, so it's basically, the (cython) LoadElement has a member link_target, which is declared as a ScalarNode in the loadelement.pyi (not sure if that counts for anything besides mypy), and ScalarNode has `as_str()` in node.pyi | 12:09 |
| tristan | yet, pylint wants to whine about it, confusing :-S | 12:09 |
| tristan | benschubert, anyway of course I'll get to the bottom of it, going for dinner now, just poked you as you might happen to be familiar with the subtleties of cython + pylint | 12:21 |
| benschubert | tristan: let me have a look | 12:42 |
| *** tristan has quit IRC | 12:44 | |
| benschubert | ugh, that seems weird. I would need more than a few minutes to understand can try tonight | 12:45 |
| *** tristan has joined #buildstream | 13:42 | |
| *** ChanServ sets mode: +o tristan | 13:42 | |
| *** tristan has quit IRC | 13:58 | |
| *** benschubert has quit IRC | 13:59 | |
| *** benschubert has joined #buildstream | 13:59 | |
| tomaz | can't find this on the help, so maybe exists but it's hidden, do we have a `bst check blah.bst` to just verify that there are no errors on the yml definitions and the build graph is correct, no missing elements? | 14:28 |
| juergbi | tomaz: no, but `bst show` is not that far off: https://gitlab.com/BuildStream/buildstream/-/issues/638 | 14:31 |
| tomaz | hm, ok. | 14:32 |
| gitlab-br-bot | juergbi opened MR !2077 (juerg/push->master: Pull missing artifacts in `bst artifact push`) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/2077 | 15:57 |
| *** santi has quit IRC | 16:51 | |
| *** jward has quit IRC | 17:37 | |
| *** benschubert has quit IRC | 17:37 | |
| *** jward has joined #buildstream | 17:49 | |
| *** benschubert has joined #buildstream | 18:04 | |
| benschubert | juergbi: 'stage_source', does it need the _deterministic_umask() still? Or is that handled by casd now? At https://gitlab.com/BuildStream/buildstream/-/blob/master/src/buildstream/element.py#L1553 | 18:25 |
| benschubert | If it is not required, it seems to fix my test_pull_access_right test? | 18:26 |
| benschubert | (it is entered concurrently, and thus does: 'save old umask, set new umask, save 'old (but new)' umask, set new umask, set old umask, set 'old (but new)' umask, #crash | 18:27 |
| benschubert | that's the sequence of events that makes it fail | 18:27 |
| benschubert | I think we can retire _deterministic_umask except when checking out sources? | 18:28 |
| benschubert | (I'm running the whole test suite to validate this hypothesis. If it works I think I'll make a MR just with this change) | 18:28 |
| tomaz | when I do a `bst checkout something.bst` that will checkout the entire build for the element, correct? (if I build a library, the library will be in the checkout) | 18:30 |
| tomaz | people, I need help on a specific library, QtX11Extras, my source is here. gitlab.com/tcanabrava/kde-build-meta.git | 18:48 |
| tomaz | the problem, when I download and compile the source as specified on the elements.bst, locally, it generates the Qt5X11ExtrasConfig.cmake, and the library | 18:49 |
| tomaz | when I try to use the element on other element, as depends, cmake complains that it can't find Qt5X11ExtrasConfig.cmake | 18:49 |
| tomaz | when I `bst checkout qt5/qt5x11extras.bst` , it does not gives me the files I'm looking for | 18:50 |
| tomaz | (no library, no cmake) - I'm not sure I did anything wrong, and this element follows the same spec as the other Qt libraries I did, all of the other ones work. | 18:51 |
| juergbi | benschubert: for staging sources as part of the build job, staging will be virtual/cas without storing any permissions (besides executable bit), so umask should not be relevant. source checkout might indeed be the only place where umask is relevant (although ideal behavior may also not be clear) | 19:50 |
| juergbi | I also can't think of a case where we want the host umask to influence any job code, i.e., maybe setting deterministic umask for the whole scheduler run would make sense | 19:52 |
| juergbi | tomaz: yes, checkout will checkout the full artifact (and even runtime dependencies by default, can be disabled with --deps none) | 19:55 |
| juergbi | do you see the library/cmake files after `bst checkout` but it's missing when used as a dependency? | 19:56 |
| juergbi | or are the files missing in both cases? | 19:56 |
| benschubert | juergbi: I can try setting it for everything too indeed. I'll try this and update my MR then if that works, thanks! | 19:56 |
| juergbi | benschubert: as it's a race condition, do you have a reliable trigger? | 19:57 |
| benschubert | re-running the test 10 times? | 19:57 |
| benschubert | which we could do in the test too (pytest.repeat(10)) | 19:58 |
| benschubert | juergbi: we've got quite a few times where umask is actually taken from the host, though I am not sure why any of this is done | 20:11 |
| benschubert | and concerning the utils.deterministic_umask, it's actually used only once for this staging of source | 20:11 |
| benschubert | which goes through casd anyways nowadays | 20:12 |
| benschubert | So, I'll just remove that call | 20:12 |
| juergbi | benschubert: hm, if I'm reading the code correctly, temp_staging_directory is not really used in that method | 20:14 |
| juergbi | it's assigned to import_dir but import_dir is always reassigned before it's used, afaict | 20:14 |
| benschubert | that's true | 20:15 |
| juergbi | so import_dir is always a cas-based directory, afaict | 20:15 |
| benschubert | So... it seems to be an artifact of the past. I'll see if removing all of it makes any test unhappy | 20:15 |
| juergbi | however, vdirectory may be filebased | 20:15 |
| benschubert | how so? Isn't is always a "virtual" directory and thus going through casd? | 20:16 |
| juergbi | vdirectory is a regular local filesystem path for `bst source checkout` and junctions (in the loader, I want to replace this) | 20:16 |
| benschubert | Or am I missing something? | 20:16 |
| benschubert | right | 20:16 |
| benschubert | but we're never actually creating it there right? | 20:17 |
| juergbi | creating what? | 20:17 |
| benschubert | the directory? | 20:18 |
| benschubert | If I understand correctly, it ends up either: | 20:18 |
| benschubert | - cas based | 20:18 |
| benschubert | - using safe_copy, which should preserve permissions anyways, thus ignoring the umask? | 20:18 |
| benschubert | (or safe_link, which does the same) | 20:19 |
| juergbi | I think that's correct for files. umask might be used for subdirectories, though | 20:20 |
| juergbi | however, if it is, we should fix the inconsistency | 20:20 |
| juergbi | and actually, preserving permissions is probably a bad idea | 20:21 |
| juergbi | as the permissions will typically be 0444 for CAS object files | 20:21 |
| juergbi | and for files with executable bit, we explicitly set it to 0755 in _filebaseddirectory.py | 20:22 |
| juergbi | so lots of inconsistencies that we need to fix | 20:22 |
| benschubert | ``` | 20:23 |
| benschubert | if os.path.lexists(old_dir): | 20:23 |
| benschubert | dir_stat = os.lstat(old_dir) | 20:23 |
| benschubert | mode = dir_stat.st_mode | 20:23 |
| benschubert | if stat.S_ISDIR(mode) or stat.S_ISLNK(mode): | 20:23 |
| benschubert | os.makedirs(new_dir) | 20:23 |
| benschubert | yield (new_dir, mode) | 20:23 |
| benschubert | else: | 20:23 |
| benschubert | raise UtilError("Source directory tree has file where " "directory expected: {}".format(old_dir)) | 20:23 |
| benschubert | ``` | 20:23 |
| benschubert | Directories are also handled there? | 20:23 |
| benschubert | Yeah :/ I'm not sure I grasp the full extent of the changes | 20:23 |
| benschubert | What would you recommend as way forward? Removing the umask now and have an issue before bst2.0? Untangle that mess before the scheduler changes? | 20:24 |
| juergbi | that's from utils._copy_directories() which we don't use for CAS -> local filesystem | 20:24 |
| benschubert | Right, but then for cas, -> local, isn't cas the one creating the directories? | 20:25 |
| benschubert | So it would be unaffected by the umask of BuildStream, since already running as a separate process | 20:25 |
| juergbi | no, it's using the FileBasedDirectory.import_files() code path, afaict. which is completely handled in Python to support things like file results with conflicts etc. (even though we don't need it there) | 20:26 |
| juergbi | I don't think removing the deterministic umask context manager in that function makes things worse with regards to overall consistency | 20:26 |
| juergbi | so if the tests pass (without remaining race conditions) with that change, I think that's fine for that branch | 20:27 |
| juergbi | we should definitely open an issue about this, though, if we haven't already | 20:27 |
| benschubert | So you'd rather not have this as a separate MR? Ok, then I can do that | 20:27 |
| benschubert | Looking for 'umask', I can't see any issues opened | 20:28 |
| juergbi | benschubert: if it's not necessary to get the tests fixed in your MR, you don't have to include it in your MR at all, imo | 20:28 |
| benschubert | ah it's necessary for my MR | 20:28 |
| benschubert | so I'll add it there | 20:28 |
| benschubert | was just wondering if I should separate | 20:28 |
| benschubert | ok, so I'll open an issue for us to validate all our umasks and such before 2.0 tomorrow | 20:29 |
| juergbi | the full fix of these inconsistencies definitely make sense as separate MR. but for the minimal fix for the threaded tests, I'm fine either way | 20:29 |
| benschubert | ok! | 20:29 |
| benschubert | thanks a lot | 20:31 |
| tomaz | juergbi: the files are missing in both cases. | 20:45 |
| tomaz | on my *local* build: | 20:45 |
| tomaz | ~/Downloads/qtx11/qtx11extras-everywhere-src-5.15.1 ⌚ 19:42:06 | 20:45 |
| tomaz | $ fd | rg libQt5X11Extras.so | 20:45 |
| tomaz | lib/libQt5X11Extras.so | 20:45 |
| juergbi | maybe it's not installing into install-root for some reason | 20:46 |
| tomaz | on my *checout*: | 20:46 |
| tomaz | /data/coiso2 ⌚ 21:45:43 | 20:46 |
| tomaz | $ fd | rg libQt5X11 | 20:46 |
| tomaz | - nothing. | 20:46 |
| juergbi | with 'local' build you mean a manual build outside BuildStream? | 20:46 |
| tomaz | yes, same tarball, but outside of buildstream. | 20:46 |
| juergbi | tomaz: what about other qt libraries in the same buildstream project? qtbase etc. are fine? | 20:48 |
| juergbi | ah, qtbase is not a qmake element | 20:48 |
| juergbi | have you built any other qmake elements and checked those or might it be a general issue with qmake elements? | 20:49 |
| tomaz | I haven build all the KDE5 elements on the kf5/ folder, at least one of them depend on qt5wayland, and that works. | 20:50 |
| tomaz | (all of the elements besides 2 that depends on Qt5X11Extras) | 20:50 |
| tomaz | the definition for QtWayland is here: https://gitlab.com/tcanabrava/kde-build-meta/-/blob/master/elements/qt5/qtwayland.bst | 20:50 |
| tomaz | the definition for Qt5X11Extras is here: https://gitlab.com/tcanabrava/kde-build-meta/-/blob/master/elements/qt5/qtx11extras.bst | 20:51 |
| tomaz | virtually identiccal | 20:51 |
| tomaz | questions (because I'm too tired to stay on the computer now) - how do I check (or where do I check) the build inside `bst shell qt5x11extras` - the folder that I go when the command finishes is empty. | 20:52 |
| juergbi | tomaz: do you see any files in the qtx11extras checkout (with --deps none)? | 20:52 |
| tomaz | sec. | 20:52 |
| tomaz | no, empty folder. | 20:53 |
| juergbi | if the artifact is empty, both checkout and shell directory will be empty | 20:53 |
| juergbi | the build log of qtx11extras may provide some insight | 20:53 |
| juergbi | especially the install part. is it installing into the proper directory? | 20:53 |
| tomaz | how do I get the log? I'm running `bst build qt5/qt5x11extras.bst` again, but it finalizes without errors. | 20:54 |
| tomaz | cached aad759f8f4484747e56e95d5f2dfbb2bdc6b6a93e6c1c781d10334a517727721 qt5/qtx11extras.bst | 20:54 |
| tomaz | but empty cache | 20:55 |
| benschubert | you can run bst artifact log qt5/qt5x11extras.bst to get access to the logs | 20:55 |
| tomaz | thanks. | 20:55 |
| juergbi | I think that's not available yet in bst 1.x | 20:55 |
| benschubert | ah right | 20:55 |
| benschubert | sorry, been too long in 2.0 land | 20:55 |
| tomaz | yeah, bst 1.5.1 here. | 20:55 |
| juergbi | check ~/.cache/buildstream/logs/kde/... | 20:55 |
| tomaz | how can remove the cached artifact (1.5x) in a way that's not grepping for the hash of the artifact in the folder? | 21:01 |
| tomaz | because the "build" logs are currently misleading: make[1]: Nothing to be done for 'install'. | 21:02 |
| tomaz | so I'll try to nuke the build folder, try to build from scratch, perhaps it was a `ctrl + c` on the wrong time that broke things. | 21:03 |
| juergbi | I don't think there is an end user command in 1.x for artifact removal | 21:03 |
| juergbi | master has it | 21:03 |
| tomaz | there's also not a way for me to `--ignore-cached-files` nor `--force-rebuild`, it seems. | 21:04 |
| juergbi | there is a directory per element for the artifact references | 21:04 |
| juergbi | manually deleting all references of an element is the simplest way to delete an artifact (reference) in 1.x | 21:05 |
| gitlab-br-bot | BenjaminSchubert approved MR !2077 (juerg/push->master: Pull missing artifacts in `bst artifact push`) on buildstream https://gitlab.com/BuildStream/buildstream/-/merge_requests/2077 | 21:05 |
| juergbi | as you can just delete that directory | 21:05 |
| juergbi | .cache/buildstream/artifacts/PROJECT/ELEMENT, iirc | 21:05 |
| tomaz | yeah, and I have a 5400 rpm disk, just doing find . -name "hash" is taking more than 5 minutes to return | 21:05 |
| juergbi | you shouldn't need ot search for the hash for this | 21:06 |
| tomaz | my artifacts folder does not have a PROJECT/ELEMENT, it has a cas/ extract/ tmp/ | 21:06 |
| juergbi | ah, does it have PROJECT/ELEMENT in cas/refs/heads/ | 21:07 |
| juergbi | ? | 21:07 |
| juergbi | it's been a while since 1.x | 21:07 |
| tomaz | yeah, just deleted. | 21:07 |
| tomaz | but deleting it made nothing, running bst build just returned that it was cached | 21:08 |
| juergbi | and it didn't pull the artifact, did it? | 21:08 |
| tomaz | [00:00:06][][] SUCCESS Build | 21:08 |
| tomaz | Pipeline Summary | 21:08 |
| tomaz | Total: 216 | 21:09 |
| tomaz | Session: 0 | 21:09 |
| tomaz | Pull Queue: processed 0, skipped 0, failed 0 | 21:09 |
| tomaz | Fetch Queue: processed 0, skipped 0, failed 0 | 21:09 |
| tomaz | Build Queue: processed 0, skipped 0, failed 0 | 21:09 |
| tomaz | I"m exausted, I'll gather all of the information I can and continue tomorrow. | 21:09 |
| tomaz | juergbi: thanks for trying to help :) | 21:09 |
| juergbi | ok. yw | 21:10 |
| tomaz | tomorrow I"ll try again. | 21:10 |
| juergbi | it's a bit difficult remembering the bst 1.x details | 21:10 |
| tomaz | juergbi: I understand, the problem is that bst2 is not released yet :) | 21:10 |
| juergbi | indeed | 21:10 |
| tomaz | so I can't depend on it for kde-build-meta (nor freedesktop-sdk) | 21:10 |
| tomaz | and those missing commands (--refresh-build, --delete) are something that I'm missing. | 21:11 |
| juergbi | understandable | 21:13 |
| benschubert | juergbi: https://gitlab.com/BuildStream/buildstream/-/merge_requests/1982/diffs?commit_id=3ef6795779cd1f8d5c6daef063a81a6a254b525b does that seem correct to you? | 21:18 |
| benschubert | I've ran the test a dozen of time and didn't have any problem | 21:18 |
| juergbi | ok, looks reasonable to me | 21:20 |
| benschubert | thanks! I'll tackle the asserts tomorrow/end of week so we might finally get that one in | 21:25 |
| *** lchlan has quit IRC | 21:52 | |
| *** jjardon has quit IRC | 21:52 | |
| *** jward has quit IRC | 21:52 | |
| *** lchlan has joined #buildstream | 21:53 | |
| *** jjardon has joined #buildstream | 21:55 | |
| *** ChanServ sets mode: +o jjardon | 21:55 | |
| *** jward has joined #buildstream | 21:55 | |
| *** jjardon has quit IRC | 21:56 | |
| *** jjardon has joined #buildstream | 21:57 | |
| *** ChanServ sets mode: +o jjardon | 21:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!