| *** tristan has quit IRC | 02:33 | |
| *** tristan has joined #buildstream | 02:45 | |
| *** adds68 has quit IRC | 03:22 | |
| *** adds68 has joined #buildstream | 03:22 | |
| *** bochecha has joined #buildstream | 05:30 | |
| *** uajain has joined #buildstream | 05:55 | |
| bochecha | oik, so I just figured out why Python feels the need to recompile all bytecode at startup inside of a BuildStream shell (or after a checkout, ...) | 06:31 |
|---|---|---|
| bochecha | when building Python modules, the first 4 bytes of the generated bytecode file are a magic number corresponding to the Python version, the next 4 bytes are the timestamp of **the source .py file** | 06:32 |
| bochecha | after building and committing to ostree, ostree resets the timestamp of all files to 0 | 06:32 |
| bochecha | as a result, in a bst shell, the source .py file has a 0 timestamp, but the .pyc file still contains the timestamp of the .py file before it was committed to ostree | 06:33 |
| bochecha | I see two possibilities to get out of this: | 06:34 |
| bochecha | 1. as pre-configure-commands, reset the timestamp of everything to 0 | 06:34 |
| bochecha | 2. as post-install-commands, set those 4 bytes in all .pyc files to 0 | 06:34 |
| bochecha | anybody can think of a better way? | 06:34 |
| *** tristan has quit IRC | 06:45 | |
| gitlab-br-bot | buildstream: issue #94 ("BuildStream could be more helpful when build Python code") changed state ("opened") https://gitlab.com/BuildStream/buildstream/issues/94 | 07:19 |
| *** jonathanmaw has joined #buildstream | 08:19 | |
| *** ssam2 has joined #buildstream | 08:30 | |
| adds68 | Sorry i messaged your messages last night tristan | 08:30 |
| adds68 | missed* | 08:31 |
| adds68 | I shall recompile and check to see if configure writes to that directory | 08:31 |
| adds68 | If i do need to re-compile with gir1-1.2, where do i get the source for that? | 08:31 |
| *** laurenceurhegyi has left #buildstream | 08:32 | |
| *** tlater has joined #buildstream | 08:35 | |
| *** laurenceurhegyi has joined #buildstream | 08:50 | |
| adds68 | *sigh* OSTree documentation is poor | 09:23 |
| ssam2 | which parts ? | 09:41 |
| adds68 | ssam2, any form of installation information | 09:43 |
| adds68 | ssam2, i'm trying to install the latest ostree from source, but buildstream is still saying my ostree version is out of date | 09:44 |
| adds68 | even though introspection is enabled in ./configure | 09:44 |
| tlater | adds68: Did you follow tristan's advice from yesterday? | 09:44 |
| tlater | :( | 09:44 |
| adds68 | tlater, i've re-run and checked for introspection yes, but still no luck | 09:45 |
| ssam2 | buildstream's setup.py is complaining about the version, you mean ? | 09:45 |
| adds68 | ssam2, yea, which tristan said is do with gir1.2-ostree being out dated | 09:45 |
| ssam2 | the setup.py seems to be doing `from gi.repository import OSTree; if OSTree.YEAR_VERSION < REQUIRED_OSTREE_YEAR: ...` | 09:46 |
| ssam2 | you can try running that code manually in `python3` to debug it. | 09:46 |
| adds68 | ssam2, that's where the issue lies, my OSTree Gobject isnt showing the YEAR_VERSION | 09:46 |
| ssam2 | hmm | 09:46 |
| adds68 | when i import it via python3 | 09:46 |
| ssam2 | what prefix is it in? you might need to set GI_TYPELIB_PATH | 09:46 |
| ssam2 | if it's not installed in /usr | 09:46 |
| adds68 | ssam2, it installs it to /usr/local/lib | 09:47 |
| tlater | Specify that path anyway, /usr/local is sometimes (usually) not included | 09:48 |
| tlater | Also, remove the gir1.2-ostree you installed with apt to avoid loading the old version by accident - will help sorting out issues like these. | 09:49 |
| adds68 | tlater, specify that where sorry? | 09:49 |
| tlater | GI_TYPELIB_PATH | 09:49 |
| ssam2 | distros aren't really consistent on whether /usr/local works or not, in my experience | 09:49 |
| ssam2 | so yeah, GI_TYPELIB_PATH=/usr/local/lib64/girepository-1.0 may be needed | 09:50 |
| tlater | Debian doesn't like it ;) | 09:51 |
| ssam2 | (double check that path actually has files in, it might install somewhere else like lib/) | 09:51 |
| adds68 | that location is set on my .bashrc | 09:52 |
| adds68 | just checked and it contains buildstream files | 09:53 |
| adds68 | but still no luck when i install | 09:53 |
| adds68 | :( | 09:53 |
| ssam2 | buildstream files? that directory should contain .typelib files | 09:53 |
| adds68 | oops sorry yea, .typelib | 09:53 |
| adds68 | *tab completion confused me * | 09:54 |
| ssam2 | hmm... this still sounds like a gobject introspection issue | 09:54 |
| tlater | Asking the obvious, but have you sourced your .bashrc since? | 09:54 |
| adds68 | tlater, yea, still no luck | 09:58 |
| tlater | Did you accidentally build the same 2 year old version of ostree or something? x) | 09:59 |
| adds68 | my ostree is version 2017.12 | 09:59 |
| adds68 | wehn i do ostree --version | 10:00 |
| adds68 | But it seems the GObject files are not compiled for the newer version | 10:00 |
| tlater | adds68: Have you removed the debian package since? If you have, you shouldn't be able to import the ostree things in the first place. | 10:12 |
| adds68 | tlater, yea ostree is removed from apt | 10:13 |
| adds68 | So ostree works only when i run the make install | 10:13 |
| tlater | Hm, shot in the dark (you'll have to recompile after), but can you still run those imports if you remove the ostree stuff you installed to /usr/local? | 10:15 |
| tlater | ^ Not sure it's worth trying | 10:15 |
| adds68 | tlater, yep removing those removes ostree | 10:18 |
| tlater | ... | 10:18 |
| * tlater is dumbfounded | 10:18 | |
| adds68 | heh, i feel lost now | 10:18 |
| tlater | Wait, did you run the setup.py code? | 10:18 |
| adds68 | tlater, as i try and install buildstream via pip again? | 10:19 |
| tlater | adds68: Nah, just run the bit of setup.py that tries to import ostree in a shell | 10:20 |
| adds68 | tlater, yea and it still gives me the same error, which is shouldnt right? Shouldn't it detect ostree isnt installed? | 10:20 |
| tlater | ^ | 10:20 |
| tlater | Do you still run it in the same virtualenv? | 10:20 |
| adds68 | tlater, yea | 10:21 |
| tlater | Create a new one and see if that changes things :) | 10:21 |
| * tlater hopes adds68 kept the build directories | 10:24 | |
| adds68 | tlater, yea it reports OSTree not found now | 10:24 |
| tlater | Whoo! So it was all virtualenv's fault in the first place. Just make install again and it should work | 10:24 |
| tlater | (And probably make a new virtualenv again) | 10:24 |
| adds68 | tlater, OSTree can't be found? | 10:25 |
| adds68 | Processing /home/adamjones/buildstream | 10:26 |
| adds68 | Complete output from command python setup.py egg_info: | 10:26 |
| adds68 | OSTree not found | 10:26 |
| adds68 | BuildStream requires OSTree >= v2017.8 with Python bindings. Install it using your package manager (usually ostree or gir1.2-ostree-1.0). | 10:26 |
| tlater | After reinstalling and creating a new virtualenv? | 10:26 |
| adds68 | Yea =/ | 10:28 |
| adds68 | fresh env with site-packages | 10:28 |
| tlater | What happens if you run that outside of any virtualenv? | 10:29 |
| adds68 | tlater, weirdly the same thing | 10:29 |
| adds68 | even though OSTree is there | 10:29 |
| tlater | echo $GI_TYPELIB_PATH, see if that's perhaps wrong | 10:30 |
| adds68 | tlater, /usr/local/lib/girepository-1.0 | 10:30 |
| tlater | And there are files in that dir? | 10:32 |
| adds68 | yea .typelib | 10:32 |
| * tlater is pretty sure it's just that env variable not being correct/enough | 10:32 | |
| tlater | If you're willing to mess with things, try installing it to /usr instead of /usr/local. | 10:33 |
| ssam2 | this is super weird, it really sounds like a g-i issue still | 10:34 |
| ssam2 | i can come look in person in a little while if you like | 10:34 |
| adds68 | ssam2, that would be a great relief! :D | 10:35 |
| ssam2 | ok, give me a few minutes | 10:36 |
| adds68 | solved! I'm a .rc noob | 10:45 |
| tlater | adds68: Did you actually forget to source it in the end or simply set it weirdly? | 10:46 |
| adds68 | tlater, i forgot to export it :( | 10:47 |
| tlater | Ah, right, and because you ran things in a subshell that didn't work x) | 10:49 |
| *** tristan has joined #buildstream | 12:14 | |
| tristan | bochecha, strip-commands I think is where that really belongs | 12:15 |
| tristan | bochecha, thats where by default we do the stripping of debug symbols, seems appropriate | 12:16 |
| bochecha | tristan: the pyc stuff? | 12:18 |
| tristan | I guess yeah | 12:19 |
| tristan | seems like post processing of installed stuff, maybe shouldnt be called strip commands | 12:19 |
| tristan | eh | 12:19 |
| bochecha | would we put it globally for every modules? | 12:19 |
| tristan | but generally the post- stuff is not meant to be declared by plugins (it's really there so that users can append something without overriding) | 12:20 |
| bochecha | I talked with Alex in #flatpak, I need to write down the details in the ticket, because there are some corner cases I hadn't thought about | 12:20 |
| tristan | I'm thinking of removing the post- commands in favor of a declarative way for appending/prepending/overriding arrays/lists | 12:20 |
| bochecha | being able to prepend/append would be nice, not just for this by the way | 12:20 |
| bochecha | I've been wanting to add more split-rules in my project;conf, and had to copy-paste the default ones | 12:21 |
| tristan | and nah, we can define the strip-commands in both python elements (I think it's pip and a setuptools element also exists) | 12:21 |
| bochecha | that's not enough, though | 12:21 |
| bochecha | I'm building Python itself, which is an autotools element | 12:21 |
| bochecha | and many python modules are built without pip/distutils, some use the autotools | 12:22 |
| tristan | right so there is this huge thread on buildstream-list https://mail.gnome.org/archives/buildstream-list/2017-September/msg00000.html | 12:22 |
| bochecha | oh, I didn't even know there was a mailing-list | 12:22 |
| tristan | which discusses some last minute changes we want to make (we changed a lot from the initial proposal) | 12:23 |
| tristan | before declaring stable 1.0 | 12:23 |
| tristan | yeah, we haz mail :) | 12:23 |
| tristan | ok so; we do have the same issue with building python itself I guess ? | 12:23 |
| tristan | i.e. it will install C compiled stuff from cpython (which we'll also want to strip debug symbols), and it will install python files | 12:24 |
| bochecha | yes | 12:24 |
| bochecha | in fact, most of the startup penalty I'm getting is from the .py/.pyc files coming with Python itself | 12:24 |
| tristan | right now in the base project.conf defaults (in data/projectconfig.yaml), we declare the strip commands as a variable | 12:25 |
| tristan | (which is pretty C library specific I guess), and then we reuse that variable in various build elements as the strip-commands | 12:25 |
| tristan | I suppose something similar could be done | 12:25 |
| tristan | where we declare that thing; as a variable (kind of like a macro)... and use it in the python elements | 12:26 |
| tristan | And then an autotools element which wanted to install python files, like when building cpython, could additionally append that macro | 12:26 |
| tristan | to its own strip commands, and then it's defined in one place but used where needed ? | 12:27 |
| bochecha | that would work, yes | 12:27 |
| bochecha | we could even go further, and make the current strip-commands be variables as well, so that it's easier to reuse and compose with the new pyc one | 12:28 |
| tristan | I suppose I should be worried about an uncontrollable accumulation of stuff being pre-defined, but we already have it for strip-commands and this does seem sensible | 12:28 |
| tristan | right, so we have %{strip-binaries} defined as a variable | 12:29 |
| tristan | and then autotools uses it http://buildstream.gitlab.io/buildstream/elements/autotools.html#module-elements.autotools | 12:29 |
| tristan | to implement strip-commands | 12:29 |
| tristan | bochecha, I think that means it's already a variable, if that's what you mean ? | 12:30 |
| bochecha | that's what I meant, I had just missed that strip-binaries existed :) | 12:30 |
| bochecha | so yes, we could define a fix-pyc-timestamps variable, add it to strip-commands for the distutils/pip elements, and let people add it themselves to other types of elements | 12:31 |
| bochecha | or just add it to the default project.conf, dunno | 12:31 |
| tristan | bochecha, well yeah that's the thing; the variable itself would have to be defined in the project.conf defaults (data/projectconfig.yaml) | 12:33 |
| tristan | which is where %{strip-binaries} is defined | 12:33 |
| tristan | there's not really another venue to make variables accessible to all element types | 12:33 |
| bochecha | yes, I meant **add** it to strip-commands by default in project.conf, or just in the distutils/pip elements | 12:33 |
| tristan | (hence my reference above, that maybe I should worry about that... but then, I dont feel like worrying about that right now :-/) | 12:34 |
| tristan | Ahhh | 12:34 |
| tristan | Nah lets not do that | 12:34 |
| tristan | also it's not possible :) | 12:34 |
| bochecha | why not? | 12:35 |
| tristan | bochecha, the fact that there exists a 'strip-commands' is the domain of BuildElement classes, actually the base project configuration has no knowledge of that | 12:35 |
| tristan | while it's possible to do overrides of element specific configuration (i.e. composite this dictionary against all 'config' sections of 'autotools' elements), what that config might be, is not known | 12:36 |
| *** jude has quit IRC | 12:37 | |
| bochecha | tristan: is it expected that when a strip-commands fail, the build still succeeds? | 13:19 |
| tristan | bochecha, nope, the command should exit 0 itself if it's okay to fail | 13:31 |
| bochecha | hmm, it's the `find ... -exec ...` | 13:31 |
| bochecha | the command being executed for each file failed on the first file | 13:32 |
| tristan | yeah, that doesnt have a history of every failing | 13:32 |
| bochecha | so it didn't get run on the other files | 13:32 |
| tristan | oh does that happen ? | 13:32 |
| tristan | interesting | 13:32 |
| bochecha | and then BuildStream happily moved along and reported SUCCESS | 13:32 |
| tristan | right, the strip commands -exec sh -ec ... also | 13:33 |
| tristan | bochecha, do you suspect this is an issue with %{strip-binaries} ? | 13:34 |
| tristan | I dont think so | 13:34 |
| bochecha | dunno, I only had it with my new command (the pyc one) | 13:34 |
| tristan | bochecha, notice that %{strip-binaries} has an explicit exit 0 clause | 13:34 |
| tristan | in a given if about en ELF binary | 13:34 |
| bochecha | I had made a typo in the python code being run, so it failed with a SyntaxError | 13:34 |
| bochecha | which failed the whole `find ...` | 13:35 |
| bochecha | but not the build | 13:35 |
| tristan | Ah well, theres nothing really to do about this I think; if you mean buildstream should have caught the problem | 13:35 |
| bochecha | yeah, I'm surprised it didn't | 13:35 |
| tristan | The BuildElement has a policy of running shell commands with -e | 13:35 |
| tristan | which means the first command to fail in that shell will abort further processing immediately | 13:36 |
| tristan | but this is find happily reporting 0 exit status I suppose | 13:36 |
| bochecha | indeed, it reports 0 | 13:38 |
| bochecha | thank you, find! | 13:38 |
| tristan | :) | 13:39 |
| bochecha | hahahahahahaha | 14:10 |
| bochecha | of course after 30 minutes of build Python, the .pyc files don't have their timestamp fixed | 14:10 |
| bochecha | because the fix-pyc-timestamps variable calls the `python` command, which isn't in the buildroot when I'm building Python itself -_- | 14:10 |
| tristan | heh | 15:19 |
| tristan | maybe redefine %{python} for that case... or: PATH=%{install-root}/%{prefix}/bin:${PATH} %{funky-python-timestamp-thingy} | 15:21 |
| * tristan assumes bochecha came to some conclusion for that anyway :) | 15:22 | |
| bochecha | PATH isn't enough | 15:22 |
| gitlab-br-bot | push on buildstream@cross_platform (by Tristan Maat): 20 commits (last: Simplify a bit of code) https://gitlab.com/BuildStream/buildstream/commit/f40ae33b52197241880d7e6399bfd0e28c4ee468 | 15:22 |
| bochecha | because running /buildstream/install/usr/bin/python means Python won't find its own libpython.so | 15:22 |
| bochecha | it will search in /usr/lib, not in /buildstream/install/usr/lib | 15:23 |
| bochecha | my conclusion is that I either need to set LD_LIBRARY_PATH, or do the timestamp mangling in another element coming after the one building Python | 15:23 |
| bochecha | and then I started doing something else :P | 15:24 |
| *** semanticdesign has joined #buildstream | 15:27 | |
| jonathanmaw | has there been a change to the way host-arches are handled? I'm getting a surprising new backtrace | 15:29 |
| jonathanmaw | https://pastebin.com/1VejnNxK | 15:30 |
| *** semanticdesign has quit IRC | 15:34 | |
| bochecha | although I wonder if it's really a good idea to have Python modify its own bytecode which it has already loaded (in order to be able to do that modification) | 15:36 |
| *** semanticdesign has joined #buildstream | 15:37 | |
| *** semanticdesign has quit IRC | 15:53 | |
| *** semanticdesign has joined #buildstream | 15:53 | |
| *** semanticdesign has quit IRC | 15:54 | |
| *** semanticdesign has joined #buildstream | 15:55 | |
| gitlab-br-bot | buildstream: merge request (cross_platform->master: Cross platform) #81 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/81 | 16:06 |
| *** semanticdesign has quit IRC | 16:29 | |
| *** adds68 has quit IRC | 16:42 | |
| *** adds68 has joined #buildstream | 16:46 | |
| *** jonathanmaw has quit IRC | 16:48 | |
| tristan | bochecha, myeah I see the concern | 16:49 |
| tristan | bochecha, actually; I think you could do better with just something like `dd` | 16:49 |
| tristan | avoid loading everything in memory and rewriting the whole file, just to zero out a few bytes | 16:50 |
| tristan | and also avoid invoking python to do it | 16:50 |
| bochecha | I'm not sure how to use dd to do that, right now I've changed the thing to only write the 4 bytes with Python and also not load the pyc files when starting the interpreter | 16:52 |
| bochecha | I'll look at dd | 16:52 |
| tristan | we do some pretty off the wall stuff with dd here: http://buildstream.gitlab.io/buildstream/elements/x86image.html#module-elements.x86image | 16:53 |
| tristan | heh | 16:53 |
| tristan | but yeah it'll let you write x bytes of value y at position z | 16:54 |
| tristan | or, no, an input stream, so you could use if=/dev/zero | 16:55 |
| bochecha | yeah, I just found out about conv=notrunc and seek=N/skip=N | 16:55 |
| bochecha | so basically, `dd if=/dev/zero of=/path/to/foo.py ibs=1 count=4 seek=4 conv=notrunc` should do it | 16:57 |
| tristan | I think so, I'm not sure if you also need obs=1 | 16:58 |
| tristan | It looks inferable though :) | 16:58 |
| tristan | I think bs=1 | 16:58 |
| tristan | for when ibs = obs | 16:59 |
| *** ssam2 has quit IRC | 17:00 | |
| *** tlater has quit IRC | 17:00 | |
| bochecha | tristan: by the way, did you see I had added some more thoughts to the ticket after talking with Alex? | 17:04 |
| tristan | bochecha, yeah I didnt want to get too deep into really thinking something through (it's now 2am so probably not much more from me...) | 17:06 |
| tristan | at face value, it does look like we would want to avoid such hackeries | 17:06 |
| tristan | but it would be worth knowing what the original specific cases triggered those hacks to occur | 17:06 |
| tristan | seems strange that mulitple packages install the same file; buildstream raises a warning for such overlaps | 17:07 |
| tristan | (although the warning will happen *a lot* when building a stack of stuff, on top of another existing stack, like building GNOME on a debian base) | 17:08 |
| tristan | should be reducible to nothing when building a full clean os from the bootstraps up | 17:08 |
| bochecha | oh, I thought you were in a European timezone... I'll make sure to ping you earlier from now on :) | 17:09 |
| tristan | :) | 17:13 |
| gitlab-br-bot | push on buildstream@fix-pyc-timestamps (by Mathieu Bridon): 1 commit (last: Fix timestamps in Python bytecode) https://gitlab.com/BuildStream/buildstream/commit/10541c428f8feb923f80566baf7ce8b13cdbdb37 | 17:43 |
| gitlab-br-bot | buildstream: merge request (fix-pyc-timestamps->master: WIP: Fix timestamps in Python bytecode) #98 changed state ("opened"): https://gitlab.com/BuildStream/buildstream/merge_requests/98 | 17:44 |
| *** semanticdesign has joined #buildstream | 18:26 | |
| *** jude has joined #buildstream | 19:39 | |
| *** semanticdesign has quit IRC | 19:40 | |
| *** bochecha_ has joined #buildstream | 21:01 | |
| *** bochecha has quit IRC | 21:02 | |
| *** bochecha_ is now known as bochecha | 21:02 | |
| *** jude has quit IRC | 21:13 | |
| *** tristan has quit IRC | 22:13 | |
| *** tristan has joined #buildstream | 22:14 | |
| *** bochecha has quit IRC | 22:15 | |
| *** tristan has quit IRC | 22:30 | |
| *** tristan has joined #buildstream | 22:34 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!