Repository navigation
cargo >nightly-2018-09-30 crashes on IBM POWER9 ppc64le with Fedora 29 #57345
Description
Activity
Thanks for the report. Seems very similar rust-lang/cargo#6320 that occurs on big endian...
Thanks for the report. Seems very similar rust-lang/cargo#6320 that occurs on big endian...
I saw that other issue before; the backtrace is largely different, and recompiling just works so. I am confused as to how it could not build correctly, I built mine with stable 1.30.1 and it's OK. I had to modify the source a little to allow use of experimental clippy features.
recompiling just works
And so it does for the big endian. Why I think these are similar issues is because just like in the other report the pc is somewhere around
0x100000000or some similar very-round number. Basically it feels to me that we simply embed a bad openssl for either PPC. It could be caused by an older toolchain, older openssl source or some other reason. I do agree that it is likely an infrastructure issue.The SIGIILL is happening because the program has branched somewhere that's not code. In fact it branches to 0 (relocated) which is the start of the ELF.
It seems to be coming from here:
│0x10088715c <rand_pool_acquire_entropy+124> nop │ │0x100887160 <rand_pool_acquire_entropy+128> mr r4,r3 │ │0x100887164 <rand_pool_acquire_entropy+132> beq cr3,0x1008871c0 <rand_pool_acquire_entropy+224> │ │0x100887168 <rand_pool_acquire_entropy+136> mr r4,r31 │ │0x10088716c <rand_pool_acquire_entropy+140> bl 0x100000000 │ >│0x100887170 <rand_pool_acquire_entropy+144> nopWhich is also what we see in the obdump:
00000000008870e0 <rand_pool_acquire_entropy>: 8870e0: 47 00 4c 3c addis r2,r12,71 8870e4: 20 88 42 38 addi r2,r2,-30688 ... 887158: b1 f3 ff 4b bl 886508 <rand_pool_add_begin+0x8> 88715c: 00 00 00 60 nop 887160: 78 1b 64 7c mr r4,r3 887164: 5c 00 8e 41 beq cr3,8871c0 <rand_pool_acquire_entropy+0xe0> 887168: 78 fb e4 7f mr r4,r31 88716c: 95 8e 77 4b bl 0So for some reason we have a branch to address 0, but there's nothing at address 0 and in fact there isn't even code there. Which looks like some sort of toolchain screwup.
If I build openssl and look at the objdump of
./crypto/rand/libcrypto-lib-rand_unix.oI think this matches up with this code (code-gen is not identical):230: 78 f3 c4 7f mr r4,r30 234: 78 eb a3 7f mr r3,r29 238: 01 00 00 48 bl 238 <rand_pool_acquire_entropy+0x88> 238: R_PPC64_REL24 rand_pool_add_begin 23c: 00 00 00 60 nop 240: 78 f3 c4 7f mr r4,r30 244: 01 00 00 48 bl 244 <rand_pool_acquire_entropy+0x94> 244: R_PPC64_REL24 getentropy 248: 00 00 00 60 nopSo it's the call to
getentropythat's resulted in the bad branch. And interestinglygetentropyis a weak symbol:$ nm ./crypto/rand/libcrypto-lib-rand_unix.o | grep getentropy w getentropyAnd:
extern int getentropy(void *buffer, size_t length) __attribute__((weak));That leads us to this issue:
openssl/openssl#6591Which suggests the
getentropycall requires GLIBC 2.25 or newer to work.I don't actually know where or how the rust ppc64le binaries are built. But it looks like they are built against glibc 2.17:
$ readelf -V cargo ... Version needs section '.gnu.version_r' contains 6 entries: Addr: 0x0000000000003458 Offset: 0x003458 Link: 5 (.dynstr) 000000: Version: 1 File: libdl.so.2 Cnt: 1 0x0010: Name: GLIBC_2.17 Flags: none Version: 9 0x0020: Version: 1 File: libm.so.6 Cnt: 1 0x0030: Name: GLIBC_2.17 Flags: none Version: 8 0x0040: Version: 1 File: libgcc_s.so.1 Cnt: 3 0x0050: Name: GCC_3.3 Flags: none Version: 7 0x0060: Name: GCC_3.0 Flags: none Version: 6 0x0070: Name: GCC_4.2.0 Flags: none Version: 5 0x0080: Version: 1 File: libc.so.6 Cnt: 1 0x0090: Name: GLIBC_2.17 Flags: none Version: 4 0x00a0: Version: 1 File: libpthread.so.0 Cnt: 1 0x00b0: Name: GLIBC_2.17 Flags: none Version: 3 0x00c0: Version: 1 File: ld64.so.2 Cnt: 1 0x00d0: Name: GLIBC_2.17 Flags: none Version: 2Presumably building on a system with GLIBC 2.25 or newer would fix the issue.
I have GLIBC 2.28 over here, so that's probably it, thanks for the detailed investigation
cc @jrtc27
<jrtc27> infinity0: ok I've found the problem <jrtc27> https://sources.debian.org/src/openssl/1.1.1a-1/crypto/rand/rand_unix.c/?hl=127#L286-L287 <jrtc27> on whatever old build machine upstream's cargo was linked against, there was no getentropy symbol in their libc <jrtc27> however, the linker has done stupid things <jrtc27> getting the address of getentropy for the function pointer != NULL comparison has remained, as it's loading it from the TOC (PPC's version of the GOT), and there's a dynamic relocation there to fill in the entry <jrtc27> so on stretch (or other older systems) the entry gets filled in as NULL at runtime and the condition gives false <jrtc27> but on sid, the entry is filled in with the non-NULL address for the function pointer and the condition gives true, so it then goes to actually call getentropy <jrtc27> however, the *call* to getentropy is done in the assembly as `bl getentropy`, ie not using the function pointer <jrtc27> and for whatever reason, the linker didn't create a PLT stub for that, but instead went "oh, call to non-existent weak function, that's a call to 0, ie crash" <jrtc27> so you get inconsistency <jrtc27> infinity0: I can reproduce it with jessie's binutils 2.25 using ld.bfd <jrtc27> but the linker bug appears to have been fixed by 2.28 <jrtc27> tried to find the relevant commit, but there are lots between 2.25 and 2.28 that mention undefined weak symbols, so I give up <jrtc27> anyway, point is, rust upstream just needs to not just an ancient build environmentcc @rust-lang/infra
Updating the docker images should be fairly easy, I think.
I'm not sure if it's as easy as just updating our images since I suspect that'll break other use cases in practice (we're on them for a reason after all).
Upgrading only the binutils usually doesn't break compatibility with older systems but without running it on older PPC distro you can't really know.
Where/how are the builds done currently?
If it's just a binutils upgrade that's fine to land at any time, and they're located in
src/ci/docker/*files which configures all our distribution builds.- added a commit that references this issue
on Mar 5, 2019 10 remaining items
The official builds are only updated as part of the release process. I'll nominate my PR for beta and stable, but I'm not aware of any plans for a 1.33.z point release.
Well the builds don't work at all, it's not a new release, rather a build system update? Don't change the version? I don't know if that's OK with rustup.
Our build system fetches its configuration from the source code it's building, so to get @cuviper's patch in we'd have to make an actual point release.
@pietroalbini Okay then, when (approximate) should I expect the next stable release?
You can see projected release dates on this page: https://forge.rust-lang.org/
Thank you @cuviper
PowerPC will not have working stable releases from rustup until the 11th of April 2019, unless you make a point release. I'd appreciate if you did make one, if the same thing happened on x86, you'd make one.Thanks for all the great work.
if the same thing happened on x86, you'd make one.
Certainly, but x86 bootstraps everything else -- it cannot be left broken. There is also a notion of tiered support, where i686 and x86_64 are tier 1, and other arches are tier 2, "not guaranteed to produce a working build." (but also "patches are always welcome!" as I have provided in this case.)
Since you're on Fedora, is there a reason you can't just use its rust packages?
(I'm also the package maintainer there.)@cuviper
There's no platform with tier1 support that can run without proprietary firmware, or else, old computers.Fedora packages are not nightly and installing both rustup and Fedora packages is not supported. I often have to switch between stable and nightly for diverse projects with different requirements.
@Leo-LB it's not officially supported but you can point
rustupto your system toolchain as explained here: https://github.com/rust-lang/rustup.rs#working-with-distribution-rust-packagesThis way you can make
systemtoolchain the default on and usecargo +nightly build.Reacted by Josh StoneNow there's another problem, it's that IntelliJ IDEA with Rust support wont install additional components automatically because the "system" toolchain does not support that.
But it's OK, I can code with nightly and compile with stable or nightly as needed.
Thank you @cuviper
PowerPC will not have working stable releases from rustup until the 11th of April 2019, unless you make a point release. I'd appreciate if you did make one, if the same thing happened on x86, you'd make one.Unfortunately it looks like 1.34 also exhibits this issue.
Correct, the infrastructure team declined the backport: #58986 (comment)
The current beta 1.35 and forward should be fine.
Correct, the infrastructure team declined the backport: #58986 (comment)
The current beta 1.35 and forward should be fine.
It should have been fixed for 1.34.
It's not like a binutils update requires any kind of stabilization, especially in front of something that does not work.Explained the reason why the backport to 1.34 was rejected in a comment in another PR.
hello, cargo >nightly-2018-09-30 shipped through rustup crashes on startup on ppc64le.
cargo binary compiled on my machine from sources on master or branch rust-1.32.0 runs fine
crash log
My machine is a RaptorCS Talos II, it has an IBM POWER9 processor, my system is Fedora 29 with latest updates.
I would guess this is a build environment issue on your end; please indicate how I can help, thanks a lot for your time!