DataOps & CI/CD Pipeline (Docker, Python, MySQL & GitHub Actions)

A comprehensive DataOps framework applying modern software engineering principles—containerization, automated continuous integration, and data validation—to reliable data pipelines.

---

🏗️ DataOps Workflow

flowchart LR
    A["Developer<br/>Git Commit / PR"] --> B["GitHub Actions<br/>(CI Workflow)"]
    
    subgraph CI Pipeline ["Continuous Integration & Quality Gates"]
        B --> C["Code Quality<br/>(Flake8 / Ruff Lint)"]
        B --> D["Automated Tests<br/>(Pytest Suite)"]
        B --> E["Docker Build<br/>Container Check"]
    end
    
    E -->|Deploy / Run| F["Docker Compose<br/>Environment"]
    
    subgraph Execution Layer ["Runtime & Storage"]
        F --> G["Python ETL Engine<br/>(Ingestion & Transform)"]
        G --> H["MySQL Database<br/>(Relational Warehouse)"]
    end

Loading

---

🚀 Key Highlights & Capabilities

• Containerized Development: Completely reproducible environment orchestrated via Docker Compose (isolated Python runner + MySQL service).
• Automated CI/CD Workflows: GitHub Actions pipeline triggers on every push and pull request, enforcing code quality, dependency validation, and unit tests before merging.
• Robust ETL Processing: Modular Python scripts for data ingestion, sanitization, type validation, and batch database loading.
• Relational Data Storage: MySQL database configured with automated healthchecks, volume persistence, and initialization schemas.

---

🛠️ Tech Stack & Tooling

Component	Technology	Specification
Language	Python 3.10+	Modular ETL processing
Containerization	Docker & Docker Compose	Isolated services
Database	MySQL 8.0+	Relational data layer
CI/CD	GitHub Actions	Automated test & lint workflow
Testing	Pytest	Automated unit testing
Code Quality	Flake8 / Black	PEP8 compliance

---

🏁 Quickstart & Local Execution

1. Prerequisites

• Docker & Docker Compose installed
• Git

2. Run the Environment

# Clone the repository
git clone https://github.com/diegobarbosaa/de-dataops.git
cd de-dataops

# Spin up MySQL and run the ETL pipeline
docker compose up --build

3. Run Tests

pytest tests/ -v

---

📁 Repository Structure

de-dataops/
├── .github/
│   └── workflows/
│       └── ci.yml             # GitHub Actions CI/CD workflow
├── app/
│   ├── main.py                # Pipeline entrypoint
│   ├── database.py            # MySQL connector & pool manager
│   └── transform.py           # Data cleansing & validation logic
├── tests/
│   └── test_pipeline.py       # Automated unit tests
├── Dockerfile                 # ETL container specification
├── docker-compose.yml         # Multi-container orchestration
├── requirements.txt           # Python dependencies
└── README.md