Add SOP0012 for Withdraw Dataset from Node Allele Frequency Beacon by costero-e · Pull Request #73 · GenomicDataInfrastructure/standard-operating-procedures

costero-e · 2026-04-10T14:20:49Z

Summary

Add new node-specific SOP template GDI-SOP0012_withdraw_dataset_from_node_allele_frequency_beacon.md.
Convert the reviewed draft into repository-compliant Markdown and align it with the SOP template, style guide, and glossary rules.
Add the allele frequency beacon specific withdrawal of the broader dataset-withdrawal workflow and link it from SOP0009.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New content (non-breaking change which adds new content)
Modified content (non-breaking change which modifies existing content)
Breaking change (fix or feature that would cause existing functionality to not work as expected)

Motivation and Context

Issue #65 requested a node-specific SOP describing how a node allele frequency beacon should handle dataset withdrawal. This PR adds the allele frequency beacon of that workflow as a reusable node-specific template.

References

Addresses [SOP Request] Dataset Withdrawal from Node Allele Frequency Beacon #65

Changes Introduced

Added sops/node-specific/GDI-SOP0012_withdraw_dataset_from_node_allele_frequency_beacon.md
Updated sops/european-level/GDI-SOP0009_dataset-withdrawal.md to link to the new nodes' allele frequency beacon SOP from step 8.7
Updated sops/README.md with the new SOP index entry
Updated CHANGELOG.md under Unreleased
Updated docs/GDI-SOP_charter.md with the new Glossary entries

Review

The SOP content is based on the reviewed draft prepared for issue #65 and converted into the repository template. In this PR, the SOP is limited to deletion of the dataset from the allele frequency beacon. Full or partial withdrawal don't apply as frequencies are calculated per dataset and statically, meaning that the removal of a dataset doesn't affect other datasets and global counts are to be made on the client side and on the fly.

Additional Notes

The SOP is added as a generic node-specific template, not as a node instance.
Auditing is to be done manually per node, until beacon provides this feature in a future release.
There is no way in the Beacon specification that allows a beacon instance to currently show a metadata response with the history of datasets that existed and were removed. We need a clearer discussion before this can be retrieved via a simple beacon query.
This PR will have conflicts with the ongoing Add SOP0011 for soft withdrawal from node FAIR Data Point #71

Checklist:

General Compliance:

My changes follow the code style of this project (GDI SOP Style Guide) and the file naming conventions of the file accessioning proposal.
I have verified that all new updated content is accessible, including checking that all external references are readable (i.e., no broken links). These may include references to external resources that should be resolvable, and internal references among SOPs.
I have properly added this PR's changes to the repository CHANGELOG.md.

Only applicable if the PR includes new, or changes to, GDI SOPs (i.e., documents at sops/):

My SOP-related changes adhere to the Generic SOP Template, including format and required fields.
I have consulted the Charter, ISM, and ORR documents to ensure compliance.
I am complying with the established procedure for SOP creations and modifications, including respecting review phases and notifying needed contributors for reviews.

costero-e · 2026-04-10T14:53:11Z

I have added @AlesCUMSI to the review as he is an active beacon user and I think he can provide some help. Thank you, @AlesCUMSI. Let me know if you don't have the time now to do it and I will choose someone else.

AlesCUMSI

Looks good overall 👍 I’ve left a few minor suggestions regarding wording, clarity, and consistency.

AlesCUMSI · 2026-04-13T07:42:41Z

+```bash
+docker exec mongoprod /bin/bash -c 'mongosh beacon -u <user> -p <password> --authenticationDatabase admin --eval "db.datasets.deleteMany({\"id\": \"<id>\"})"'
+```
+


You have redinex in both hard deletions, but not here. Is this correct?

Yes, because in the soft one, the variants are never removed 👍

If variants are never removed, then can the effects of the presence of that data propagate to other datasets?

In other words, the effective deletion of the dataset would not be real, as it would not change the AFs?

The AFs are statically calculated, not on the fly. There's confusion about this, because we are not allowing progressive updates. Everything works as a single package. This is what was agreed in GDI, as for now. So, no, no propagation possible.

costero-e · 2026-04-13T14:24:48Z

I have now finished adding @AlesCUMSI suggestions. Thanks for the useful feedback @AlesCUMSI , it helped a lot!

M-casado

Great work @costero-e . Also thanks to @AlesCUMSI , as many of my comments went in line with his.

I have some comments that I left in line, and some general comments here:

If the soft deletion is defined as data being retained, isn't it inconsistent to use db.datasets.deleteMany to delete the dataset from mongoDB? I'm not familiar with AF Beacon, so maybe it's not, but the command sounds like we're deleting data.
The dataset_config.yml referenced as per the documentation (in main) here contradicts the isDeprecated in this SOP. Either isTest and isSynthetic are not the only values, or... something is off
Verification in step 4 doesn't distinguish deletion type: it says to verify "datasets and variants" queries for both types, but if I went through the soft-deletion path, I only removed dataset records (not variants).
Not sure if this is meant in the Beacon spec, but isDeprecated flag applies to both types of deletion without explanation. For hard-deletion everything is already deleted, so would we be adding a flag to a dataset that is gone? Is this the way to audit changes in the Beacon?

M-casado · 2026-04-13T10:14:08Z

 | API              | Application Programming Interface                                       |
 | BEN              | Beneficiary                                                             |
 | CC               | Carbon Copy (used in communications to send a copy to others)           |
+| CRG              | Centre for Genomic Regulation                                           |


Missing "AF" for Array Frequency

In both the charter and the SOP0012

If i might make a suggestion based on @M-casado comment (If the soft deletion is defined as data being retained, isn't it inconsistent to use db.datasets.deleteMany to delete the dataset from mongoDB? I'm not familiar with AF Beacon, so maybe it's not, but the command sounds like we're deleting data.): What if we nothing is deleted soft-delete, mainteiner just needs to remove dataset id from beacon/permissions/datasets/datasets_permissions.yml file? If my memory serves me correct, this prevents beacon from returning dataset (and g-variants connected to it)?

Regarding the soft-deletion, if we only modify the datasets_permissions.yml file, the dataset will continue appearing in the /datasets endpoint, which for beacon means: we have this dataset and you will be able to retrieve it depending on your permissions. I think this is not the situation we want, exactly. Deleting just the datasets entry will block any beacon response to return anything from this dataset. This is a very small record, that only contains metadata for the dataset, and I think is just what a soft-deletion pursues, deletion of metadata regarding a dataset (at the end is soft-deletion, something, very minimal, has to be deleted). What hard-deletion does is deleting the much bigger amount of variants, which can't be rolled back with a simple insert. For me, the approach I did is correct.

M-casado · 2026-04-13T15:15:17Z

+
+| Step identifier | When | Who |
+| :-------------- | :--------------------------------------- | :-------------------------------------- |
+| `4`             | After successful completion of Step 3 | Node aggregated beacon maintainer |


The issue I have with the current steps, is that I think we're missing one that is for failure. Otherwise, steps 2 redirect you to step 3 on failure of withdrawal, and then you come here and it's phrased as if the withdrawal was successful, and to verify the records are not appearing. Hence why I would have a step of failure in the diagram/procedure, to be explicit with the logic

I added this: After ⏩Step 3.1 or ⏩Step 3.2, maybe with this is not necessary to add a new Step just for failure?

…le-frequency-beacon.md Co-authored-by: Marcos Casado <mcasado@ebi.ac.uk>

…le-frequency-beacon.md Update on actor of the step 1 Co-authored-by: Marcos Casado <mcasado@ebi.ac.uk>

…le-frequency-beacon.md VHD report clarification Co-authored-by: Marcos Casado <mcasado@ebi.ac.uk>

…le-frequency-beacon.md Co-authored-by: Marcos Casado <mcasado@ebi.ac.uk>

AlesCUMSI · 2026-05-03T10:30:38Z

+| Author     | Liina Nagirnaja             | Beacon Manager                                   | CRG                                   |
+| Author     | Jordi Rambla                | Beacon Product Owner                             | CRG                                   |
+| Author     | Oriol López-Doriga Sagalés  | Beacon Developer                                 | CRG                                   |
+| Author     | Aleš Čep                    | Task 4.3 member                                  | CRG                                   |


I am not sure why my name is added as Author. This must be mistake. If any, maybe change it to Reviewer. Also, my organization is not CRG, it is a University Of Maribor (UM)

Apologies @AlesCUMSI , changed it now!

M-casado · 2026-06-04T09:50:18Z

@costero-e - I was wondering if after you addressed @AlesCUMSI's and my feedback on this PR, it is ready for the approval round of the OC and SDPC in GDI. I've seen a few comments still pending, but some are now outdated.

Let me know your thoughts so that we can continue where we left

costero-e · 2026-06-05T07:27:05Z

Hi @M-casado, I re-reviewed the SOP, executing all the commands and I think that the comments pending mostly are outdated and the only thing I see a bit hanging is the FAIR principles one (find datasets that were removed previously from the beacon) that is dependant on a greater solution of the specification. I could do a workaround but would be only implementation-specific. If we can move on without this, then, the SOP is ready.

M-casado · 2026-06-05T08:30:38Z

Great @costero-e - As an author of other SOPs myself, I'm aware that there's always room for improvements, but I agree it's best we get the GDI OC/SDPC involved already, as the SOP framework is built around continuous revision. Especially with the feature requiring a spec change.

Great work, once again!

M-casado · 2026-06-05T08:34:48Z

Just communicated the request to the OC/SDPC

costero-e · 2026-06-05T08:38:18Z

Great @costero-e - As an author of other SOPs myself, I'm aware that there's always room for improvements, but I agree it's best we get the GDI OC/SDPC involved already, as the SOP framework is built around continuous revision. Especially with the feature requiring a spec change.

Great work, once again!

Perfect then. And thanks to you for your invaluable contributions!

omllobet · 2026-06-22T15:50:30Z

Hi Oriol,

Thanks for the corrections.

In the section about "Remove dataset permission", I think it would be useful to remind the operator to save the relevant information for auditing purposes before removing the dataset from datasets_permissions.yml, rather than only describing the removal step, specially for controlled access.

Also, in the diagram, nodes K and L appear to have the same label, which results in "Remove dataset permissions" being shown twice.

costero-e · 2026-06-23T07:51:32Z

Hi @omllobet,

Thanks a lot for your useful feedback. I have addressed now the identified issues.

Let me know if you find anything else that is not making sense.

Best!

omllobet · 2026-07-01T15:37:33Z

I approve this SOP as a member of the OC

omllobet · 2026-07-09T12:26:15Z

I approve this SOP as a member of the SDPC

M-casado · 2026-07-29T15:59:37Z

Status summary:

We are still pending another approver (besides @omllobet). @GabiRinck is listed as an approver in the RR section of the SOP0012, which could account for it. Have you officially approved this SOP as a member of both committees, Gabi?
@costero-e please add @omllobet as an approver in the table within SOP0012.
The conflicts are minimal and can easily be resolved. Essentially it's just merging with what the SOP0011 added to the shared files.

GabiRinck · 2026-07-30T08:49:55Z

Thanks, Marcos,
Oscar approved on behalf of the OC & SDPC
Kjell (@kjellp) volunteered as 2nd approver for the OC & Nejc (@NNadizar) as 2nd approver for the SDPC - apologies for not tagging you as reviewers before. Many thanks!

M-casado · 2026-07-30T08:52:00Z

@costero-e - Please add the approvers with their roles to the table in the SOP before merging

costero-e · 2026-07-30T10:21:02Z

Done @M-casado

…le-frequency-beacon.md Co-authored-by: Marcos Casado <mcasado@ebi.ac.uk>

costero-e added 5 commits April 10, 2026 15:40

adding GDI-SOP0012 markdown

0b9e33a

updating GDI-SOP0009 alongside README and CHANGELOG

8d3733c

adding some forgotten aspects to the af beacon SOP

c7d3054

added glossary to charter file and other SOP fixes

310919c

addressing GDI-SOP0009 url links

c1198a3

costero-e requested review from AlesCUMSI, GabiRinck and M-casado April 10, 2026 14:50

AlesCUMSI approved these changes Apr 13, 2026

View reviewed changes

costero-e added 3 commits April 13, 2026 16:12

addressing some mistakes after Ales review

c8a7f94

addressing some mistakes after Ales review

92de254

addressing some mistakes after Ales review

cca456f

removing duplicated datasets from path file

1b46543

M-casado requested changes Apr 13, 2026

View reviewed changes

costero-e and others added 14 commits April 14, 2026 10:54

some changes addressed from Marcos review

3e0c91d

Update sops/node-specific/GDI-SOP0012_withdraw-dataset-from-node-alle…

32ece07

…le-frequency-beacon.md Co-authored-by: Marcos Casado <mcasado@ebi.ac.uk>

adding fixes from Marcos review

871910c

Update sops/node-specific/GDI-SOP0012_withdraw-dataset-from-node-alle…

6186dac

…le-frequency-beacon.md Update on actor of the step 1 Co-authored-by: Marcos Casado <mcasado@ebi.ac.uk>

Update sops/node-specific/GDI-SOP0012_withdraw-dataset-from-node-alle…

46f2a64

…le-frequency-beacon.md VHD report clarification Co-authored-by: Marcos Casado <mcasado@ebi.ac.uk>

adding fixes from Marcos review

b40875c

Update sops/node-specific/GDI-SOP0012_withdraw-dataset-from-node-alle…

49bff2d

…le-frequency-beacon.md Co-authored-by: Marcos Casado <mcasado@ebi.ac.uk>

updating SOP and steps

8f14e10

updating SOP and steps

bb9012b

updating SOP and steps

4be3b37

updating SOP and steps

98fd1ef

updating SOP and steps

5999d2b

updating SOP and steps

02795e2

fixing CHANGELOG name for the SOP

eabf3c0

adding new line for Step 8.1

d14d967

AlesCUMSI reviewed May 3, 2026

View reviewed changes

adding Ales correct role and organisation

3e45b16

costero-e mentioned this pull request Jun 10, 2026

Add SOP0014 Add datasets metadata to node beacon #75

Open

10 tasks

costero-e added 2 commits June 22, 2026 11:43

fixing table and flowchart

3825bb0

addressed flowchart for step D to A

6712742

addressed flowchart and added auditing at removing datasets permissions

7b0f779

GabiRinck requested review from NNadizar and kjellp July 30, 2026 08:43

adding reviewers

8356ec6

M-casado reviewed Jul 30, 2026

View reviewed changes

Comment thread sops/node-specific/GDI-SOP0012_withdraw-dataset-from-node-allele-frequency-beacon.md Outdated

Update sops/node-specific/GDI-SOP0012_withdraw-dataset-from-node-alle…

78c63ce

…le-frequency-beacon.md Co-authored-by: Marcos Casado <mcasado@ebi.ac.uk>

Uh oh!

Conversation

costero-e commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Types of changes

Motivation and Context

References

Changes Introduced

Review

Additional Notes

Checklist:

Uh oh!

costero-e commented Apr 10, 2026

Uh oh!

AlesCUMSI left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

costero-e commented Apr 13, 2026

Uh oh!

M-casado left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

costero-e Apr 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

M-casado commented Jun 4, 2026

Uh oh!

costero-e commented Jun 5, 2026

Uh oh!

M-casado commented Jun 5, 2026

Uh oh!

M-casado commented Jun 5, 2026

Uh oh!

costero-e commented Jun 5, 2026

Uh oh!

omllobet commented Jun 22, 2026

Uh oh!

costero-e commented Jun 23, 2026

Uh oh!

costero-e commented Apr 10, 2026 •

edited

Loading

costero-e Apr 15, 2026 •

edited

Loading