[release/2.1] Bug fixes for Derecho site configs

[climbfuji]

Description

Stop-gap measure for release/2.1 so that it supports MPAS-JEDI, UFS, and other applications. For release/2.1, we stick with unloading ncarenv and loading the original crayenv; to make this work with the UFS context and the way they unload and load modules, we must revert to the default cray-mpich version 8.1.29 that is loaded when crayenv is loaded.

For develop, a different approach will be taken that attempts to use the butchered ncarenv environment.

Dependencies

None. This PR goes with LINK TO UFS PR MISSING

Issues addressed

Issues with UFS using spack-stack-2.1.1 on Derecho, recorded in A UFS PR FOR WHICH I NEED TO FIND THE LINK

Applications affected

All supported applications on Derecho

Systems affected

Derecho

Testing

• CI: Note whether the automatic tests (GitHub actions tests that run automatically for every commit) pass or not
  • GitHub actions CI tests pass
  • GitHub actions CI tests do not pass (provide explanation)
  • GitHub actions CI tests skipped (provide explanation if necessary)
• New tests added: List and describe any new tests added to GitHub actions
  • ...
• Additional testing: Add information on any additional tests conducted
  • ...

Checklist

• This PR addresses one issue/problem/enhancement or has a very good reason for not doing so.
• These changes have been tested on the affected systems and applications.
• All dependency PRs/issues have been resolved and this PR can be merged.
• All necessary updates to the documentation (spack-stack wiki) will be made when this PR is merged