Skip to content

Download script is changed to add all quarters of data - #2115

Open
Krishnam24maheshwari wants to merge 5 commits into
datacommonsorg:masterfrom
Krishnam24maheshwari:new_branch
Open

Download script is changed to add all quarters of data#2115
Krishnam24maheshwari wants to merge 5 commits into
datacommonsorg:masterfrom
Krishnam24maheshwari:new_branch

Conversation

@Krishnam24maheshwari

@Krishnam24maheshwari Krishnam24maheshwari commented Jul 28, 2026

Copy link
Copy Markdown

This PR resolves the deletions problems caused by 13-quarter rolling window logic as per the API we are using to download data.

Changes Made in brazil_download_script.py:
1. Replaced the 13-quarter rolling window logic with get_available_periods() and different API.
2. Fixed the baseline start period explicitly to 202201 (Q1 2022) onwards through the latest available quarter.
3. Applied this updated period range across all four dataset categories (Average_Real_Income, Mass_Income, Population_Economic_sector, and Employment_And_Unemployment_Labor_Force), preserving all 2022 records and resolving false deletion validation errors.

PR Checklist -
https://docs.google.com/spreadsheets/d/1RD4U94SM2mLcjSZRLjJHVliF737fScBH4rsDU_Pu_hk/edit?gid=1086229089#gid=1086229089

@google-cla

google-cla Bot commented Jul 28, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the Brazil SIDRA IBGE download script, replacing the Selenium-based browser automation with direct API calls using the requests library and reshaping the data into Excel files using pandas. The review feedback recommends improving performance, security, and robustness by using a global requests.Session with connection pooling and retries, re-enabling SSL verification, and adding explicit guard checks to safely parse the API responses.

Comment thread statvar_imports/brazil_sidra_ibge/brazil_download_script.py Outdated
Comment thread statvar_imports/brazil_sidra_ibge/brazil_download_script.py Outdated
Comment thread statvar_imports/brazil_sidra_ibge/brazil_download_script.py Outdated
Comment thread statvar_imports/brazil_sidra_ibge/brazil_download_script.py
Krishnam Maheshwari and others added 4 commits July 28, 2026 12:23
Added a newline at the end of the file for proper formatting.
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Added a robust session setup for HTTP requests with retries and connection pooling. Updated functions to utilize the new session for API calls.
@pulkeet-wq

Copy link
Copy Markdown
Contributor

Few action items:

  1. Update README files
  2. Add try except blocks specifically while downloading the data
  3. Add proper comments to the code
  4. Add goldens

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants