Upgrading Josh

[christian-schilling]

I have been seing some things happening in josh-sync like pr #38, where @Kobzol suggested to move to the new cli because josh-sync is re-implementing things we already added upstream.
Now I set out to contribute that myself while having some time over easter but then realized there is a bigger problem here:
There was one breaking change in Josh about handleing trivial merge commits. While there is a supported migration path, it is hard to adopt since it would require some subtrees to use the old version and others the new version, at least till of them are upgraded (on all branches).
I don't expect breaking changes link this in Josh to be common, but it still got me thinking...

Things would be a lot easier if the configuration for all subtrees was centralized in one place. It's semantics are anyway tied to the "current repo setup" and not really live on the same time axis as the code itself.
Places to store is could be a ref/branch here in rust-lang/josh-sync or rust-lang/rust. (could even be on the main branch of "josh-sync" if the tool is not intended to use outside of rust-lang)

Then when doing syncs, josh-sync could get the config from there, all it would need is the "what subtree am i in" information passed to it.

Any thoughts on this? @Kobzol, @RalfJung

[RalfJung]

To me, this very much feels like repo configuration that should live in the repo. Having it elsewhere just causes gnarly questions around versioning and review.

The usual way to deal with migrations is to make them opt-in in the per-repo config. What's the problem with that approach?

[christian-schilling]

Ah this is kind of two topics in one:
A) How to deal with the upgrade/migration
B) What is the scope of the config? In particular the filter.

So for A):
Yes if the config is per commit then a "two step migration" would be needed. Like putting some flag in the config. Definetly doable. But depending on the answer to B) it might not be necessary.

For B):
These thoughts apply independently of the upgrade problem.
There are 3 scopes of the config I can think of:

• Per commit
• Per repo
• Global

I'd argue it is in fact global, or maybe per repo but certainly not per commit. But storing it in the branch itself makes it per commit.
So why do i think that?
The filter describes the relationship of r-l/r to (for example) r-l/miri commits. This relationship is (must be!) the same for all commits, even for those old ones that don't even have a josh-sync.toml. So in case anyone wanted to branch of any of those (and have corresponding synced branch with upstream) the config would need to be replicated there.
Also if the filter was changed it would mean (potentially) all miri shas change (meaning force push, not desired). So even though it is versioned in the branch, it can never be changed in the branch. So why version it there?
It also means if it was at some point decided to go for a filter change with force push, the config would need to be retroactivly updated in the whole history, since otherwise looking at any commit before the change would imply a wrong filter. That is even less desirable since the old commits that would need to be amended are already in r-l/r upstream.
So effectively the only valid configuration is the one in the head of the main branch of the subtree repo. That's a global.

Also potentially someone might want to create another subtree repo rooted in the same place in the rust tree. Let's say a version of miri with some files excluded. Or with linearized history. At that point there is another problem: With the current setup, the josh-sync.toml is also synced to the rust tree. Then two overlaping subtrees would want the same file in the upstream have different content. Why should one of them be special? Yes one could pick another name, but that seems like a workaround. And would also have the same problem: The new config would only exist "from some point onwards" even though it in fact applies to "the whole history".
It all comes down to: The relationship between the two brances is not a property of either, but an entiry of it's own.

So that leaves repo scope or global scope.
Global scope would mean putting one branch that only has those config files for all subtrees in a shared place (ref in r-l/r or here is this repo). That would make centralized automation easy.
Repo scope would mean having a "config branch" in each subtree repo. Then the sync-tool would look at that and use it for all commits in the real branch. Makes distributed maintance/automation easier.

Either of those seem potentiallly good to me. Maybe they almost don't differ in implementation 🤔

[RalfJung]

It is not per-commit, but the LICENSE will also never change and yet we have it in-tree. That's the most natural place IMO, and it is where we need the config for pulling and pushing. We have other similar config in the repo too, like triagebot.toml. That naturally means all changes of this config go through the Miri review process etc -- all behaving exactly as we want it.

I see no advantage to using magic branches for this purpose, and I can imagine various downsides (not least of which the fact that this is hard to discover).

It also means if it was at some point decided to go for a filter change with force push, the config would need to be retroactivly updated in the whole history

No, the file reflects the config at the time. No need to update anything.

Also potentially someone might want to create another subtree repo rooted in the same place in the rust tree. Let's say a version of miri with some files excluded. Or with linearized history.

As Rust project I think we have zero interest in such a repo, so I don't see why this would be a concern. Remember that in our mind, https://github.com/rust-lang/miri/ is the authoritative source for Miri and it is where Miri development happens, and josh is just a tool to get that git tree integrated into the rustc tree. That means overlap between subrepos is impossible.

[christian-schilling]

It is not per-commit, but the LICENSE will also never change and yet we have it in-tree.

Not common or likely but LICENSE can change and does apply to the tree. So that must be in tree no question.

That's the most natural place IMO, and it is where we need the config for pulling and pushing.

This is essentally the config of an extra remote. Even git itself does not store that in-tree. git-submodule does, but in a wired way: It makes a copy of the remotes during git submodule init and then keeps using that config regarless of what happens with the in-tree config.

We have other similar config in the repo too, like triagebot.toml. That naturally means all changes of this config go through the Miri review process etc -- all behaving exactly as we want it.

Reading the triagebot docs... It states "triagebot.toml in the root of the default branch". So when triagebot operates, it does not take the config from the branch or commit it operates on, but rather what ever is found at the tip of the default branch.
What I'm realizing: The ref does not matter. Whether it's "tip of default branch" (what triagebot does) or "tip of special config branch" (what i proposed as "repo scope")...the only thing that does matter is "take what's at the tip of a defined branch". So yes, it can be the main branch. It should just always use that config and not the config of whatever happens to be checked out (unless explicitly told so for experimentation). In fact the tool should not even need a checked out working copy. 🤔

I see no advantage to using magic branches for this purpose, and I can imagine various downsides (not least of which the fact that this is hard to discover).
Yep you're right, the main branch can be the magic branch 🙈

It also means if it was at some point decided to go for a filter change with force push, the config would need to be retroactivly updated in the whole history

No, the file reflects the config at the time. No need to update anything.

That's only true if josh-sync would behave like triagebot and treat the tip of main as 'magic place for the config".
It probably does exactly that when run in gh actions, because it always checks out main, but it does not locally where any other commit could be checked out.

Also potentially someone might want to create another subtree repo rooted in the same place in the rust tree. Let's say a version of miri with some files excluded. Or with linearized history.

As Rust project I think we have zero interest in such a repo, so I don't see why this would be a concern. Remember that in our mind, https://github.com/rust-lang/miri/ is the authoritative source for Miri and it is where Miri development happens, and josh is just a tool to get that git tree integrated into the rustc tree. That means overlap between subrepos is impossible.

Someone else might have intrest in that..but true, whoever that is would need to figure out thier own solution. So in that sense the authorative source is a special case..

So back to the how to upgrade: Since "tip of subtree main" cannot be changed for all subtrees at once, a way to distinguish old vs new config format is needed. (there is no need to use an old version of josh for the old format, it just needs to adjust the filter and then the new josh should still produce the same output).

Something like config_version = 2 🤔

[RalfJung]

So when triagebot operates, it does not take the config from the branch or commit it operates on,

Triagebot does not operate on a branch or commit, it operates on an issue or PR.

It probably does exactly that when run in gh actions, because it always checks out main, but it does not locally where any other commit could be checked out.

Locally there is no reason to expect a josh pull/push on an old commit to do anything sensible. Those operations implicitly act on the current commit. It seems entirely natural to me that that commit would then also configure how the operation behaves.

So back to the how to upgrade: Since "tip of subtree main" cannot be changed for all subtrees at once, a way to distinguish old vs new config format is needed. (there is no need to use an old version of josh for the old format, it just needs to adjust the filter and then the new josh should still produce the same output).

Something like config_version = 2 🤔

Since it's specific to the filter, I'd call it filter_version or so, but -- yeah, something like that seems reasonable to me.

[Kobzol]

I think that the current setup works well for us - josh-sync.toml is useful for CI (which runs in the subtree repos, so it is the easiest to access it from that repo directly) or for local operations, and those are also performed on the subtree (and not on the parent repo); at least that is how we do it today.

While upgrading Josh is generally painful when there are breaking changes, I think that adding a new optional version field to the config is easy, and would allow us to use different Josh versions or different ways of interpreting the filter in the config without breaking backwards compatibility with old versions of josh-sync or the josh-sync.toml config file.

• Subtrees that run josh-sync on CI always install the version from main. So if we do a BC break, josh-sync has to be able to distinguish subtrees with the old config, and use old Josh version and logic for it, and repos with the new config, and use new Josh and logic for it. In that case we will be able to migrate the subtrees incrementally, and once that is done, we can ditch the old Josh logic in josh-sync.
• users who run josh-sync locally should always have the latest main branch of the subtree checked out when doing Josh operations. If they have old config, then either old josh-sync, or new josh-sync, will work. But if the subtree has already migrated to the new version, then if the user has old josh-sync, it would probably do something wrong (because we don't use deny_unknown_fields for the config, so old josh-sync will ignore the new config field, rather than erroring out). But it should be immediately obvious that something is wrong, and the user will then have to update josh-sync.

[christian-schilling]

I created a PR for the upgrade: #48
The idea here here is that by setting compat options the behaviour stays the same for now so no versioning of the config is needed yet.
That can be done as next step.

[Kobzol]

We have the filter version now, so I think that in general the way to upgrade Josh going forward is "solved".