ci: enable merge queue and run Miri only in the queue

[blaginin]

• Closes MIRI jobs consume a lot of GH runner action time #11045
• closes Use a merge queue #8422

Add merge queue to arrow-rs and move miri to run there (and not on every push)

[alamb]

Merge up to get the fix for audit failure

[alamb]

Thank you @blaginin -- I will be honest that this looks much more complicated than I was expecting

Is there some reason not to just change MIRI to run on pushes to main and merge_queue? That way we would save them running on PRs. It would still run about 2x on each PR (though sometimes the merge_queue one could be shared)

[blaginin]

Thanks for the review. I was trying to copy the approach from comet:

• ci: move the job routing policy out of ci.yml expressions and into compute-changes.py datafusion-comet#5850
• ci: add a Required Checks aggregator job so main can have a required status check datafusion-comet#5842
• ci: move the Spark 3.4/3.5/4.0, Iceberg, macOS and benchmark suites behind a merge queue datafusion-comet#5843

specifically, stuff like this is a bit tricky:

[blaginin]

That's like check-ci-config.py in Comet.

[blaginin]

That's like compute-changes.py in Comet.

[Jefffrey]

i wonder if we should try simplifying our CI routing first; for example, group parquet-variant checks into just parquet, and fix some triggers for parquet (i see theres a trigger for any arrow-avro changes but i dont think theres actually a dependency in code for this?)

i do agree this routing logic feels a bit heavy, especially given (as far as i understand) in comet its because they have a lot of heavy CI tasks so they need this granular dispatch, whilst for us here its mainly miri (and maybe the arrow integration) that are heavy

• feel free to correct me if im wrong here, not as familiar with comets setup

with datafusion, did it not have this routing logic in the first place (before merge queue was implemented there)?