Spack build vs suite build reproducibility

[blimlim]

Background

We've been struggling to get the Spack build for CM3 to reproduce with the suite build. Previous investigation identified differences in compiler flags, preprocessor flags, and included libraries. Modifying the compilation in the suite build to remove these differences resulted in bitwise identical outputs from the two builds.

While differences from flags like -march=skylake-avx512 -mtune=skylake-avx512 -unroll made sense, it was confusing that the spack build's inclusion of the CDEPS dwav library and the presence of the WAV_PRESENT preprocessor flag also had an effect. CM3 doesn't use the data wave component and the code under WAV_PRESENT is not called in by CM3, so we wouldn't expect anything to change with their inclusion. @MartinDix recommended we try to understand where the change in answers are coming from in case they indicated a bug.

When adding in/removing the data wave component, differences surprisingly showed up in the very first timestep in several moisture related variables in the atmosphere, propagating to other components from there:

Summary:

The WAV_PRESENT preprocessor flag and inclusion of the cdeps-dwav (data wave) library were a bit of a red herring. The differences instead came from:

1. The CMakeLists.txt file used in the spack build places several CMEPs components after the UM in the linking arguments, while the suite build places the UM last
2. The relative location of the CMEPs components and the UM affects the order of the UM library and ESMF in the linker call. This in turn affects the order of the UM library and the standard math library -lm in the linker call.
3. The order of the UM library and -lm in the linker arguments affects whether math functions are picked up from the standard C library or the intel optimised math library libmf
4. Including cdeps-dwav only had an apparent effect as I'd placed it after the UM in the linker arguments during the investigation, changing the position of -lm

Details:

1. libum-atmos.a's location in the linker arguments

Including cdeps-dwav only leads to some modules being included, and a function call that doesn't run for CM3's settings. Commenting these sections out still showed answer differences depending on whether cdeps-wav was included or omitted during linking. This suggested something related to the linking as the culprit.

Comparing the linker calls from the build output:

cdeps-wav included

/apps/openmpi/4.1.7/bin/mpif90  
-qno-opt-dynamic-align  -convert big_endian -assume byterecl -ftz -traceback -assume realloc_lhs -fp-model precise -g -grecord-gcc-switches -O2 
-march=skylake-avx512 -mtune=skylake-avx512 -unroll -O0 -O0 "CMakeFiles/CM3.dir/home/565/sw6175/cylc-run/20260914-spackcomps-O0-wavpresent-medoutput-2/share/CMEPS/cesm/driver/esmApp.F90.o" 
-o access-cm3  libcesm_driver.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access3-share-git.a66762b7ddd157e6339fe6fd6c3c8ca7f1f7f20d_stable-v6sujklvhuun2kyok7g6ezzeltegorg5/lib64/libaccess-share.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/esmf-8.7.0-v5dy2pjbfattgkth24qy3qozx42lml3j/lib/libesmf.so 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access-mom6-2025.07.000-dug3mv4pntp3lqgcaxqc4qkbms7a4opa/lib64/libaccess-mom6lib.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access-generic-tracers-2025.08.000-45clnrueiwv2w7zyufyyadrgxal2rogx/lib64/libgtracers.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/fms-2025.03-u6knt6vrmyh4nrhf5c4f4kqsausq42yc/lib64/libfms_r8.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access-mocsy-2025.07.002-mvnskng64rmsha2uh2qmynja5557jnqg/lib/libmocsy.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access-cice-git.a98ca97971d1eb699bcdd18ebe7c6660361606c4_stable-fq4bjcpkj4bovpvqwojbfvtw7x5a2fpm/lib64/libaccess-cicelib.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/um-13.0-o2lbbjriuk3rmmbuw4jzd66te5y46ubl/lib/libum-atmos.a
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64_v4/gcom-8.4-4ftke6ffmz47w6upirdx7wzrwpspvofy/lib/libgcom.a -lgcom 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access3-share-git.a66762b7ddd157e6339fe6fd6c3c8ca7f1f7f20d_stable-v6sujklvhuun2kyok7g6ezzeltegorg5/lib64/libaccess-cdeps-dwav.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access3-share-git.a66762b7ddd157e6339fe6fd6c3c8ca7f1f7f20d_stable-v6sujklvhuun2kyok7g6ezzeltegorg5/lib64/libaccess-cdeps-common.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access3-share-git.a66762b7ddd157e6339fe6fd6c3c8ca7f1f7f20d_stable-v6sujklvhuun2kyok7g6ezzeltegorg5/lib64/libaccess-cmeps.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access3-share-git.a66762b7ddd157e6339fe6fd6c3c8ca7f1f7f20d_stable-v6sujklvhuun2kyok7g6ezzeltegorg5/lib64/libaccess-nuopc_cap_share.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access3-share-git.a66762b7ddd157e6339fe6fd6c3c8ca7f1f7f20d_stable-v6sujklvhuun2kyok7g6ezzeltegorg5/lib64/libaccess-share.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access3-share-git.a66762b7ddd157e6339fe6fd6c3c8ca7f1f7f20d_stable-v6sujklvhuun2kyok7g6ezzeltegorg5/lib64/libaccess-timing.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/parallelio-2.6.2-q3vcdqj6y34lpcw7um47zy5wo3pc4pjc/lib/libpiof.so 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/parallelio-2.6.2-q3vcdqj6y34lpcw7um47zy5wo3pc4pjc/lib/libpioc.so 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/netcdf-fortran-4.6.1-wolf7embv2muth6t36aqc2v5j3kkvlrv/lib/libnetcdff.so 
-L/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/netcdf-fortran-4.6.1-wolf7embv2muth6t36aqc2v5j3kkvlrv/lib -lnetcdff -lnetcdf -lnetcdf -lm 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/netcdf-c-4.9.2-4v3jp4kkkaehln6nikzd4dviynjgzgt3/lib64/libnetcdf.so 
-L/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/netcdf-c-4.9.2-4v3jp4kkkaehln6nikzd4dviynjgzgt3/lib64 -lnetcdf 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/esmf-8.7.0-v5dy2pjbfattgkth24qy3qozx42lml3j/lib/libesmf.so 
-Wl,-rpath,/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/esmf-8.7.0-v5dy2pjbfattgkth24qy3qozx42lml3j/lib 
-Wl,-rpath,/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/netcdf-c-4.9.2-4v3jp4kkkaehln6nikzd4dviynjgzgt3/lib64 
-Wl,-rpath,/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/netcdf-fortran-4.6.1-wolf7embv2muth6t36aqc2v5j3kkvlrv/lib  
-Wl,-rpath,/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/parallelio-2.6.2-q3vcdqj6y34lpcw7um47zy5wo3pc4pjc/lib 
-Wl,-rpath,/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/esmf-8.7.0-v5dy2pjbfattgkth24qy3qozx42lml3j/lib 
-L/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/esmf-8.7.0-v5dy2pjbfattgkth24qy3qozx42lml3j/lib 
-L/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/esmf-8.7.0-v5dy2pjbfattgkth24qy3qozx42lml3j/lib 
-L/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/netcdf-c-4.9.2-4v3jp4kkkaehln6nikzd4dviynjgzgt3/lib64 
-L/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/netcdf-fortran-4.6.1-wolf7embv2muth6t36aqc2v5j3kkvlrv/lib 
-L/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/parallelio-2.6.2-q3vcdqj6y34lpcw7um47zy5wo3pc4pjc/lib 
-lrt -ldl -lnetcdf -lnetcdff -lnetcdf -lnetcdf -lm -lpioc -m64 -mcmodel=small -pthread -threads -cxxlib -Wl,--no-as-needed -qopenmp 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/fortranxml-4.1.2-36icgdwfsls5jcs5bao3llennij5fkpw/lib/libFoX_dom.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/fortranxml-4.1.2-36icgdwfsls5jcs5bao3llennij5fkpw/lib/libFoX_sax.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/fortranxml-4.1.2-36icgdwfsls5jcs5bao3llennij5fkpw/lib/libFoX_wcml.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/fortranxml-4.1.2-36icgdwfsls5jcs5bao3llennij5fkpw/lib/libFoX_wkml.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/fortranxml-4.1.2-36icgdwfsls5jcs5bao3llennij5fkpw/lib/libFoX_wxml.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/fortranxml-4.1.2-36icgdwfsls5jcs5bao3llennij5fkpw/lib/libFoX_common.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/fortranxml-4.1.2-36icgdwfsls5jcs5bao3llennij5fkpw/lib/libFoX_utils.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/fortranxml-4.1.2-36icgdwfsls5jcs5bao3llennij5fkpw/lib/libFoX_fsys.a -lirng -ldecimal -lcilkrts -lstdc++ 

(the inclusion of both -O2 and -O0 was unintentional but I think the -O0 should win out as it's second)

No cdeps-wav

/apps/openmpi/4.1.7/bin/mpif90  -qno-opt-dynamic-align  -convert big_endian -assume byterecl -ftz -traceback -assume realloc_lhs -fp-model precise -g -grecord-gcc-switches -O2 -march=skylake-avx512 
-mtune=skylake-avx512 -unroll -O0 -O0 "CMakeFiles/CM3.dir/home/565/sw6175/cylc-run/20260914-spackcomps-O0-nowav-medoutput-2/share/CMEPS/cesm/driver/esmApp.F90.o" 
-o access-cm3  
libcesm_driver.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access3-share-git.a66762b7ddd157e6339fe6fd6c3c8ca7f1f7f20d_stable-v6sujklvhuun2kyok7g6ezzeltegorg5/lib64/libaccess-share.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/esmf-8.7.0-v5dy2pjbfattgkth24qy3qozx42lml3j/lib/libesmf.so
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access-mom6-2025.07.000-dug3mv4pntp3lqgcaxqc4qkbms7a4opa/lib64/libaccess-mom6lib.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access-generic-tracers-2025.08.000-45clnrueiwv2w7zyufyyadrgxal2rogx/lib64/libgtracers.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/fms-2025.03-u6knt6vrmyh4nrhf5c4f4kqsausq42yc/lib64/libfms_r8.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access-mocsy-2025.07.002-mvnskng64rmsha2uh2qmynja5557jnqg/lib/libmocsy.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access-cice-git.a98ca97971d1eb699bcdd18ebe7c6660361606c4_stable-fq4bjcpkj4bovpvqwojbfvtw7x5a2fpm/lib64/libaccess-cicelib.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access3-share-git.a66762b7ddd157e6339fe6fd6c3c8ca7f1f7f20d_stable-v6sujklvhuun2kyok7g6ezzeltegorg5/lib64/libaccess-cdeps-common.a
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/fortranxml-4.1.2-36icgdwfsls5jcs5bao3llennij5fkpw/lib/libFoX_dom.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/fortranxml-4.1.2-36icgdwfsls5jcs5bao3llennij5fkpw/lib/libFoX_sax.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/fortranxml-4.1.2-36icgdwfsls5jcs5bao3llennij5fkpw/lib/libFoX_wcml.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/fortranxml-4.1.2-36icgdwfsls5jcs5bao3llennij5fkpw/lib/libFoX_wkml.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/fortranxml-4.1.2-36icgdwfsls5jcs5bao3llennij5fkpw/lib/libFoX_wxml.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/fortranxml-4.1.2-36icgdwfsls5jcs5bao3llennij5fkpw/lib/libFoX_common.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/fortranxml-4.1.2-36icgdwfsls5jcs5bao3llennij5fkpw/lib/libFoX_utils.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/fortranxml-4.1.2-36icgdwfsls5jcs5bao3llennij5fkpw/lib/libFoX_fsys.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access3-share-git.a66762b7ddd157e6339fe6fd6c3c8ca7f1f7f20d_stable-v6sujklvhuun2kyok7g6ezzeltegorg5/lib64/libaccess-cmeps.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access3-share-git.a66762b7ddd157e6339fe6fd6c3c8ca7f1f7f20d_stable-v6sujklvhuun2kyok7g6ezzeltegorg5/lib64/libaccess-nuopc_cap_share.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access3-share-git.a66762b7ddd157e6339fe6fd6c3c8ca7f1f7f20d_stable-v6sujklvhuun2kyok7g6ezzeltegorg5/lib64/libaccess-share.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/parallelio-2.6.2-q3vcdqj6y34lpcw7um47zy5wo3pc4pjc/lib/libpiof.so 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/parallelio-2.6.2-q3vcdqj6y34lpcw7um47zy5wo3pc4pjc/lib/libpioc.so 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/netcdf-fortran-4.6.1-wolf7embv2muth6t36aqc2v5j3kkvlrv/lib/libnetcdff.so 
-L/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/netcdf-fortran-4.6.1-wolf7embv2muth6t36aqc2v5j3kkvlrv/lib -lnetcdff -lnetcdf -lnetcdf -lm 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/netcdf-c-4.9.2-4v3jp4kkkaehln6nikzd4dviynjgzgt3/lib64/libnetcdf.so 
-L/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/netcdf-c-4.9.2-4v3jp4kkkaehln6nikzd4dviynjgzgt3/lib64 -lnetcdf 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/access3-share-git.a66762b7ddd157e6339fe6fd6c3c8ca7f1f7f20d_stable-v6sujklvhuun2kyok7g6ezzeltegorg5/lib64/libaccess-timing.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/esmf-8.7.0-v5dy2pjbfattgkth24qy3qozx42lml3j/lib/libesmf.so 
-Wl,-rpath,/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/esmf-8.7.0-v5dy2pjbfattgkth24qy3qozx42lml3j/lib 
-Wl,-rpath,/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/netcdf-c-4.9.2-4v3jp4kkkaehln6nikzd4dviynjgzgt3/lib64 
-Wl,-rpath,/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/netcdf-fortran-4.6.1-wolf7embv2muth6t36aqc2v5j3kkvlrv/lib  
-Wl,-rpath,/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/parallelio-2.6.2-q3vcdqj6y34lpcw7um47zy5wo3pc4pjc/lib 
-Wl,-rpath,/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/esmf-8.7.0-v5dy2pjbfattgkth24qy3qozx42lml3j/lib 
-L/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/esmf-8.7.0-v5dy2pjbfattgkth24qy3qozx42lml3j/lib 
-L/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/esmf-8.7.0-v5dy2pjbfattgkth24qy3qozx42lml3j/lib 
-L/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/netcdf-c-4.9.2-4v3jp4kkkaehln6nikzd4dviynjgzgt3/lib64 
-L/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/netcdf-fortran-4.6.1-wolf7embv2muth6t36aqc2v5j3kkvlrv/lib 
-L/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/parallelio-2.6.2-q3vcdqj6y34lpcw7um47zy5wo3pc4pjc/lib 
-lrt -ldl -lnetcdf -lnetcdff -lnetcdf -lnetcdf -lm -lpioc -m64 -mcmodel=small -pthread -threads -cxxlib -Wl,--no-as-needed -qopenmp 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/um-13.0-o2lbbjriuk3rmmbuw4jzd66te5y46ubl/lib/libum-atmos.a 
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64_v4/gcom-8.4-4ftke6ffmz47w6upirdx7wzrwpspvofy/lib/libgcom.a -lgcom 
-lirng -ldecimal -lcilkrts -lstdc++ 

Notably the UM library libum-atmos.a appears quite early on in the list of arguments when cdeps-dwav is included, and towards the end of the list when it's omitted. Some googling suggested that the order of libraries in the linker call can affect results, as the linker will pick up missing symbols (functions?) as it encounters them from left to right. Changing the order apparently may change where it picks up certain symbols from.

Using nm -g on the resulting executables, and scrolling through a diff of the output suggests that this is in indeed happening:

- 0000000007e108f0 T cosh
+                 U cosh@@GLIBC_2.2.5

- 0000000007e138b0 T cpowf
+                 U cpow@@GLIBC_2.2.5


- 0000000007e10950 T expf
+                 U expf@@GLIBC_2.27

- 0000000007e10a10 T sinh
+                 U sinh@@GLIBC_2.2.5

These symbols are found within the text T (perhaps from a static library) when the UM appears earlier in the linker arguments, whereas they are left undefined (U) when the UM appears towards the end.

It's worth noting that in the UM library libum-atmos.a, these symbols are undefined:

nm -g libum-atmos.a | grep sinh
                 U sinh

2. Location of -lm

The above functions are all math functions, suggesting the C standard math library -lm might be relevant. -lm appears in the list -lrt -ldl -lnetcdf -lnetcdff -lnetcdf -lnetcdf -lm -lpioc -m64 in the linker arguments further above, which comes from ESMF /g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/esmf-8.7.0-v5dy2pjbfattgkth24qy3qozx42lml3j/lib/esmf.mk:

ESMF_F90LINKLIBS= -lrt -ldl -lnetcdf -lnetcdff -lnetcdf -lnetcdf -lm -lpioc

Several of the other components including dwav depend on ESMF, and so I'd guess that ESMF and the above arguments get placed after all these components in the final call to the linker.

Starting with the build including cdeps-wav and forcing an extra -lm flag to occur before the UM library, results are then identical to the build which omitted cdeps-wav:

# Compare prognostics in the first timestep restart dump:

mule-cumf 20260914-spackcomps-O0-wavpresent-medoutput-lm-before-um/share/data/History_Data/atmosa.d0001_progs 20260914-spackcomps-O0-nowav-medoutput-2/share/data/History_Data/atmosa.d0001_progs

Compared 12368/12368 fields, with 12368 matches

Hence, the results look dependent on whether -lm or libum-atmos.a appear first in the linker call.

3. Where the math functions come from:

Adding -Wl,--trace-symbol=sinh to the linker call, we can find out where the sinh function is taken from in each case.

-lm before libum-atmos.a:

/lib64/libm.so.6: definition of sinh
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/um-13.0-o2lbbjriuk3rmmbuw4jzd66te5y46ubl/lib/libum-atmos.a(can_drag_mod.o): reference to sinh
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/um-13.0-o2lbbjriuk3rmmbuw4jzd66te5y46ubl/lib/libum-atmos.a(ukca_ddepo3_ocean_mod.o): reference to sinh
//apps/intel-ct/2021.10.0/compiler/linux/compiler/lib/intel64_lin/libimf.so: reference to sinh
//apps/intel-ct/2021.10.0/compiler/linux/compiler/lib/intel64_lin/libifcoremt.so.5: reference to sinh

Here, it's being picked up in the standard C math library /lib64/libm.so.6

-lm after libum-atmos.a

/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/um-13.0-o2lbbjriuk3rmmbuw4jzd66te5y46ubl/lib/libum-atmos.a(can_drag_mod.o): reference to sinh
/g/data/vk83/prerelease/apps/spack/1.1/restricted/ukmo/release/linux-x86_64/um-13.0-o2lbbjriuk3rmmbuw4jzd66te5y46ubl/lib/libum-atmos.a(ukca_ddepo3_ocean_mod.o): reference to sinh
/apps/intel-ct/2021.10.0/compiler/linux/compiler/lib/intel64_lin/libimf.a(sinh_iface_c99.o): definition of sinh
//apps/intel-ct/2021.10.0/compiler/linux/compiler/lib/intel64_lin/libimf.so: reference to sinh
//apps/intel-ct/2021.10.0/compiler/linux/compiler/lib/intel64_lin/libifcoremt.so.5: reference to sinh

Here it's being picked up in the intel optimised math library /apps/intel-ct/2021.10.0/compiler/linux/compiler/lib/intel64_lin/libimf.a. It's also a static library, explaining the T vs U seen in the nm output further above.

4. Toy example

At first I though this meant something could be slightly off with our UM build. However, playing around with a toy example seems to point to it being expected behaviour:

myprog.F90:

program my_prog

    use funcs, only: run_sinh

    implicit none

    real start_val
    real end_val

    start_val = 0.1234509835324975234957234095834837246539284756

    call run_sinh(start_val, end_val)

    write(6, '(F40.30)'), end_val

end program my_prog

run_sinh_mod.F90

module funcs
    implicit none

    private
    public run_sinh

    contains

    subroutine run_sinh(in, out)
        real, intent(in) :: in
        real, intent(out) :: out

        out = sinh(in)
    end subroutine run_sinh

end module funcs

First compile run_sinh_mod.F90 into a library:

module load intel-compiler/2021.10.0
ifort -c -i8 -r8 run_sinh_mod.F90 
ar r run_sinh_mod.a run_sinh_mod.o

And then compile myprog.F90 into an executable, swapping around the order of -lm and run_sinh_mod.a:

ifort -c -i8 -r8 myprog.F90 

ifort -i8 -r8 -o myprog_lmfirst.exe myprog.o -lm run_sinh_mod.a -Wl,--trace-symbol=sinh
/lib64/libm.so.6: definition of sinh
run_sinh_mod.a(run_sinh_mod.o): reference to sinh

ifort -i8 -r8 -o myprog_lmlast.exe myprog.o run_sinh_mod.a -lm  -Wl,--trace-symbol=sinh
run_sinh_mod.a(run_sinh_mod.o): reference to sinh
/apps/intel-ct/2021.10.0/compiler/linux/compiler/lib/intel64_lin/libimf.a(sinh_iface_c99.o): definition of sinh

Inspecting the symbol tables:

nm -g myprog_lmfirst.exe  | grep sinh
0000000000403950 T funcs_mp_run_sinh_
                 U sinh@@GLIBC_2.2.5


nm -g myprog_lmlast.exe  | grep sinh
0000000000403990 T funcs_mp_run_sinh_
0000000000403a20 T __libm_sinh
000000000068b400 D __libm_sinh_chosen_core_func_x
0000000000403ff0 T __libm_sinh_e7
0000000000403b30 T __libm_sinh_ex
0000000000404e30 T __libm_sinh_l9
00000000004039b0 T sinh

And lastly we get different results from each executable:

./myprog_lmfirst.exe 
        0.123764791049152056423565682053
./myprog_lmlast.exe 
        0.123764791049152042545777874238

5. More details

ifort automatically adds in the intel math library -limf during linking. Adding the -v option to the final linking step prints out the actual call to ld used under the hood, which is very long but contains the following relevant sections:

-lm before run_sinh_mod.a:

 -Bstatic -limf -Bdynamic -lm run_sinh_mod.a -Bdynamic -Bstatic -lifport -lifcoremt -limf -lsvml -Bdynamic -lm

-lm after run_sinh_mod.a:

 run_sinh_mod.a ... -Bstatic -limf -Bdynamic -lm 

This post outlining the linker's construction of the symbol table with static libraries very helpful to get an idea of what's different in the above two lines.

In the first case, I think:

1. The linker doesn't have any undefined math symbols when it reaches the first -limf, and so it skips the library.
2. It then adds all exported symbols from the shared library -lm to the symbol table, in particular sinh.
3. run_sinh_mod.a requires the undefined symbol sinh, and so the linker connects it to the exported symbol from -lm.

And in the second case, I think:

1. run_sinh_mod.a requires sinh, so the linker adds this to the list of undefined symbols
2. The linker next encounters the static library limf, which exports sinh. As this is already in the list of undefined symbols, the linker adds the relevant object file to the executable.
3. When the linker then encounters the shared library lm, it skips it as the undefined symbols have already been resolved.

My understanding of all of this is quite hazy and I'm mostly guessing about

2. It then adds all exported symbols from the shared library -lm to the symbol table, in particular sinh.

as I couldn't find clear information on this online. In any case, playing around with another toy example and building bot a shared and static library from the same fortran module suggested:

ifort -o myprog.exe  static.a -L. -lshared  myprog.o

will pick up functions from the shared library, while

ifort -o myprog.exe  myprog.o static.a -L. -lshared

will pick up functions from the static library.

[blimlim]

@manodeep just wanted to get your thoughts here! Feel free to just read over the summary section. The main remaining question is whether we want the UM to use the standard library lm functions or the optimised intel math library limf functions. @MartinDix mentions that the radiation code makes use of many exponentials and so there may be a performance benefit from using the intel ones.

[manodeep]

The libm library is the gold standard for accuracy - that would be the trade-off here. Is that trade-off acceptable for the use-case?

[MartinDix]

I don't think accuracy is an issue as long as we still have the same reproducibility.

[MartinDix]

Are functions loaded from dynamic libraries in the order shown by ldd? If they are then AM3 is using the functions from libimf w/o a problem.

module load access-am3/2026.08.002
ldd $(which um-atmos.exe)
...
        libimf.so => /apps/intel-ct/2021.10.0/compiler/linux/compiler/lib/intel64_lin/libimf.so (0x000014a028f77000)
        libsvml.so => /apps/intel-ct/2021.10.0/compiler/linux/compiler/lib/intel64_lin/libsvml.so (0x000014a027948000)
        libm.so.6 => /half-root/usr/lib64/libm.so.6 (0x000014a0275c6000)