A Financial Market Data Engineering & Analytics Platform
Indonesia Mining Intelligence is a data engineering and analytics project designed to monitor Indonesian mining-related equities and global commodity markets.
The platform integrates historical stock market data and international commodity benchmarks into a centralized analytical database. It transforms raw financial data into structured indicators and interactive visualizations to support financial market monitoring and research.
The project explores how commodity price movements can provide additional context for understanding Indonesia's mining sector.
The project aims to:
- Build an end-to-end financial market data pipeline.
- Integrate Indonesian mining equities with global commodity benchmarks.
- Develop structured financial datasets for analytical consumption.
- Calculate historical performance and market indicators.
- Provide an interactive dashboard for market monitoring.
- Establish a foundation for cross-market and macroeconomic analysis.
Historical stock price data from selected Indonesian mining and energy-related companies.
Selected companies:
| Ticker | Company | Sector |
|---|---|---|
| ADRO.JK | Alamtri Resources Indonesia | Coal |
| PTBA.JK | Bukit Asam | Coal |
| ITMG.JK | Indo Tambangraya Megah | Coal |
| ANTM.JK | Aneka Tambang | Nickel & Gold |
| INCO.JK | Vale Indonesia | Nickel |
| NCKL.JK | Trimegah Bangun Persada | Nickel |
| MDKA.JK | Merdeka Copper Gold | Gold & Copper |
| BRMS.JK | Bumi Resources Minerals | Gold & Copper |
| AMMN.JK | Amman Mineral Internasional | Copper & Gold |
| MEDC.JK | Medco Energi Internasional | Oil & Gas |
The platform integrates five commodity instruments from two data providers.
| Commodity | Data Source | Frequency |
|---|---|---|
| Gold | Yahoo Finance | Daily |
| Copper | Yahoo Finance | Daily |
| Brent Crude Oil | Yahoo Finance | Daily |
| Nickel | World Bank Pink Sheet | Monthly |
| Australian Coal | World Bank Pink Sheet | Monthly |
The combination of daily and monthly observations supports both short-term market monitoring and long-term commodity trend analysis.
The analytical pipeline calculates:
- Daily and monthly returns.
- Simple Moving Averages.
- Rolling historical volatility.
- Intraday price range.
- Historical price highs and lows.
- Rule-based trend classification.
- Historical performance comparisons.
The Streamlit dashboard currently provides five main sections:
- Market Overview
- Equity Analysis
- Stock Performance Comparison
- Commodity Intelligence
- Pipeline Information
The project follows a layered data engineering workflow.
External Data Sources
|
v
Python Data Ingestion
|
v
Raw Data Storage (Parquet)
|
v
Data Processing
|
v
PostgreSQL Data Warehouse
|
v
dbt Transformation
|
v
Analytical Data Models
|
v
Streamlit Dashboard
| Component | Technology |
|---|---|
| Programming Language | Python |
| Data Collection | yfinance, Requests, Pandas |
| Data Processing | Pandas, NumPy |
| Storage Format | Parquet |
| Database | PostgreSQL |
| Data Transformation | dbt |
| Dashboard | Streamlit |
| Visualization | Plotly |
| Version Control | Git & GitHub |
The database separates source datasets and analytical models.
| Schema | Table | Description |
|---|---|---|
| public | stock_prices | Historical Indonesian equity prices |
| public | commodity_prices | Daily commodity futures observations |
| public | world_bank_commodity_prices | Monthly World Bank commodity prices |
The dbt project uses staging, intermediate, and mart layers.
Staging
stg_stock_pricesstg_commodity_pricesstg_world_bank_commodity_prices
Intermediate
int_stock_indicatorsint_commodity_indicatorsint_world_bank_commodity_indicators
Marts
mart_stock_performancemart_stock_historymart_historical_performancemart_commodity_performancemart_world_bank_commodity_performance
Current validated datasets:
| Dataset | Records | Frequency |
|---|---|---|
| Indonesian Mining Equities | 8,974 | Daily |
| Yahoo Finance Commodities | 2,910 | Daily |
| World Bank Commodities | 1,604 | Monthly |
Coverage period:
- Equity data: January 2023 – November 2026.
- Yahoo Finance commodities: January 2023 – November 2026.
- World Bank commodities: January 1960 – October 2026.
Record counts represent the current collected datasets and may change as new observations are ingested.
The project includes dedicated data quality checks covering:
- Missing observations.
- Duplicate records.
- Date consistency.
- Historical coverage.
- Indicator calculation accuracy.
The current commodity validation identified no duplicate observations or future-dated records.
All tested commodity indicators produced zero calculation mismatches within a numerical tolerance of 0.000001.
Missing historical observations are retained as NULL when source data is unavailable.
For detailed results, see:
financial-market-idr-monitoring/
|
|-- indonesia_mining/
| |-- config/
| |-- data/
| | |-- raw/
| | |-- processed/
| | |-- warehouse/
| |-- dashboard/
| | |-- app.py
| |-- docs/
| |-- src/
| | |-- ingestion/
| | |-- processing/
| | |-- database/
| |-- tests/
| |-- notebooks/
| |-- requirements.txt
|
|-- mining_analytics/
| |-- models/
| | |-- staging/
| | |-- intermediate/
| | |-- marts/
| |-- tests/
| |-- dbt_project.yml
|
|-- check_commodity_db.py
|-- check_data_quality.py
|-- check_indicator_validation.py
|-- .gitignore
|-- README.md
- Python 3.13
- PostgreSQL 16
- Git
- dbt
- Streamlit
git clone https://github.com/YOUR_USERNAME/financial-market-idr-monitoring.git
cd financial-market-idr-monitoringpython -m venv .venvActivate the environment.
Windows PowerShell:
.\.venv\Scripts\Activate.ps1pip install -r indonesia_mining/requirements.txtCreate an .env file inside the indonesia_mining directory.
Configure the following variables according to your local PostgreSQL setup:
DB_USER=your_database_user
DB_PASSWORD=your_database_password
DB_HOST=localhost
DB_PORT=5433
DB_NAME=mining_marketNever commit your actual database credentials to GitHub.
From the repository root:
streamlit run indonesia_mining/dashboard/app.pyThe dashboard will be available at:
http://localhost:8501
Additional ingestion and transformation instructions will be documented as the project setup is finalized.
- Yahoo Finance futures prices may differ from spot commodity prices.
- World Bank Pink Sheet data is monthly rather than daily.
- Australian Coal is used as a global benchmark proxy, not an Indonesian domestic coal price.
- Historical volatility and moving averages are descriptive indicators, not standalone forecasting tools.
- Mathematical validation confirms calculation consistency, not independent source accuracy.
- Commodity and equity observations come from different markets and calendars.
| Feature | Status |
|---|---|
| Indonesian mining equity ingestion | Implemented |
| Yahoo Finance commodity ingestion | Implemented |
| World Bank commodity ingestion | Implemented |
| PostgreSQL data warehouse | Implemented |
| dbt analytical models | Implemented |
| Streamlit dashboard | Implemented |
| Commodity data quality validation | Implemented |
| USD/IDR JISDOR integration | Planned |
| Commodity-equity correlation analysis | Planned |
| Rolling correlation analysis | Planned |
| Macroeconomic data integration | Planned |
| Automated pipeline orchestration | Planned |
This project is developed for educational, analytical, and portfolio purposes.
It does not provide investment advice, trading signals, or guaranteed market forecasts.
All market observations are subject to the availability, methodology, and limitations of their original data providers.