Skip to content

Feature/enhanced data export - #945

Open
rogonion wants to merge 35 commits into
testfrom
feature/enhanced-data-export
Open

rogonion wants to merge 35 commits into
testfrom
feature/enhanced-data-export

Conversation

@rogonion

Copy link
Copy Markdown
Collaborator

Changelog:

  • The bug whereby filtering by multiple countries resulted in incorrect results being shown was actually being caused by the filters being initialized with the wrong values (generated human readable filter values meant for inserting in the excel file export that concatenated the countries into a single string). The fix was to just change the filter value being used to the actual raw filter values as they were created.
  • The SAS (temporary signed azure blob url for downloading export files) url solution for addressing the 403 permissions failed as is still resulted in the same 403 permissions error. Therefore, since the backend API has got full rights against the azure blob, all export file downloads are now proxied via the nest API resulting in downloads flowing from this (internet->azure_blob) to this (internet->nest API->azure_blob). Caching and partial file fetching (using byte range) mechanisms have been added to the export file download module which allow the download to be more robust in slow/unstable connections i.e., ability to resume downloads instead of starting from scratch and re-downloading the same complete file uses the already cached browser download instead or beginning a fresh download.
  • Updated the data dialog to prompt the user to enter their email so that they can receive a data download link. Moreover, the download notifications have been much more enhanced with retry mechanisms in addition to a button that allows a user to manually retrigger checking the download export status thus can continue downloading even after the internet connection drops and comes back up as long as the retry notification and website are still open.
  • Updated the download export job caching mechanism meant to allow re-use of created downloads for export jobs by using the sorted occurrence ids that matched the filter query in conjunction with the unique client-side generated id and timestamp to generate unique hashes. The current solution now is if two or more users set the same filters that result in the same sorted occurrence ids, then the subsequent users after the first one to download will use the same downloaded file instead of starting a unique download export job.
  • Slight improvement in overall download speed by changing the database and typeorm queries from an N+1 query per row in each batch (initially it seems that for every row about 5+ additional individual queries were being ran to put the various tables' data together) to just a single query that returns the entire joined tables set of rows.

Update umami deployment and configuration by using custom Dockerfile image with legacy patching so that umami can be publicly accessible over /analytics base path and shared domain.
to validate if azure blob permissions are an isolated issue.
OccurrenceGeoJob: Adding ssl option
Adding updated download confirmation dialog.
TOD: re-enable after mailing system is merged.
Adding tunable parameters for data export.
Allow maskEmail to silently fail if node:crypto is not found.
Update doi link in email to use generated doi first and fallback to
default.
Changelog:
- Enable and add USE_SAS_URLS to address 403 azure blob download errors.
- Enhance the data export procedure so that it is more robust against
intermittent download issues.
- Fix DOI filter using wrong json args (human readable instead of raw
filter values) resulting in DOI not resolving properly.
- update github workflow so that final build on runs when merge is
created instead of during PR.
- Add openapi docs since it was already added previously.
(IN DEVELOPMENT, not complete or in use) New exports-v3 aims to improve
data export speed. Proposal is to directly write db rows into excel
without mapping rows.
To mitigate azure blob 403 permission errors for public downloads of
blobs, proxy downloads via the API so now request will be like
`internet->nest API->azure blob`.
Add cache headers and byte ranged requests so that downloads are more
robust on slow/unstable connections.
DOI would not resolve properly if a full data download was performed and
thus no filters were set resulting in infinite loading screen.
Fix formData sending 'undefined' string thus causing direct downloads to
fail. Enhance email sending module to better detect empty email
recipients.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant