This document outlines all test cases needed for DeepScraper to achieve comprehensive test coverage and address static analysis requirements. The test suite covers unit tests, integration tests, and end-to-end tests for all components.
Create jest.config.js:
module.exports = {
preset: 'ts-jest',
testEnvironment: 'node',
roots: ['<rootDir>/src'],
testMatch: ['**/__tests__/**/*.test.ts'],
collectCoverageFrom: [
'src/**/*.ts',
'!src/**/*.d.ts',
'!src/types/**',
'!src/index.ts'
],
coverageThreshold: {
global: {
branches: 80,
functions: 80,
lines: 80,
statements: 80
}
},
setupFilesAfterEnv: ['<rootDir>/jest.setup.js'],
moduleNameMapper: {
'@/(.*)': '<rootDir>/src/$1'
}
};Create jest.setup.js:
// Mock environment variables
process.env.NODE_ENV = 'test';
process.env.REDIS_HOST = 'localhost';
process.env.REDIS_PORT = '6379';
process.env.CACHE_ENABLED = 'true';
process.env.OPENAI_API_KEY = 'test-key';
// Global test timeout
jest.setTimeout(30000);
// Mock external dependencies
jest.mock('bullmq');
jest.mock('ioredis');
jest.mock('playwright');File: src/api/controllers/__tests__/scraper.controller.test.ts
describe('ScraperController', () => {
describe('scrapeUrl', () => {
// Test Case 1: Successful URL scraping
it('should successfully scrape a valid URL with default options', async () => {
// Arrange: Mock ScraperManager response
// Act: Call scrapeUrl with valid URL
// Assert: Response contains expected data structure
});
// Test Case 2: Invalid URL handling
it('should return 400 for invalid URL format', async () => {
// Test URLs: '', 'not-a-url', 'javascript:alert(1)'
});
// Test Case 3: Different extraction formats
it('should return content in requested format (HTML/Markdown/Text)', async () => {
// Test each format option
});
// Test Case 4: LLM extraction with schema
it('should extract structured data when schema is provided', async () => {
// Mock OpenAI response
// Verify extracted data matches schema
});
// Test Case 5: Timeout handling
it('should handle timeout gracefully', async () => {
// Mock delayed response
// Verify timeout error response
});
// Test Case 6: Cache hit scenario
it('should return cached response when available', async () => {
// Mock cache hit
// Verify no scraping occurs
});
// Test Case 7: Browser action execution
it('should execute browser actions in sequence', async () => {
// Test click, scroll, fill actions
});
// Test Case 8: Proxy configuration
it('should use proxy when configured', async () => {
// Verify proxy settings are applied
});
// Test Case 9: Error propagation
it('should properly propagate scraper errors', async () => {
// Test network errors, blocked URLs, etc.
});
// Test Case 10: Response headers
it('should include proper CORS and security headers', async () => {
// Verify response headers
});
});
});File: src/api/controllers/__tests__/crawler.controller.test.ts
describe('CrawlerController', () => {
describe('crawl', () => {
// Test Case 1: Basic crawl initiation
it('should initiate crawl with valid URL', async () => {
// Mock WebCrawler and queue
// Verify job creation and response
});
// Test Case 2: Streaming crawl mode
it('should use streaming mode when useMapDiscovery is true', async () => {
// Mock CrawlKickoffService
// Verify streaming response
});
// Test Case 3: Traditional crawl mode
it('should use traditional mode when useMapDiscovery is false', async () => {
// Mock queue service
// Verify queue job creation
});
// Test Case 4: URL pattern filtering
it('should apply includePaths and excludePaths filters', async () => {
// Test regex pattern matching
});
// Test Case 5: Depth and limit constraints
it('should respect maxDepth and limit parameters', async () => {
// Verify crawl boundaries
});
// Test Case 6: Subdomain handling
it('should handle allowSubdomains option', async () => {
// Test subdomain inclusion/exclusion
});
// Test Case 7: Robots.txt compliance
it('should respect robots.txt unless ignored', async () => {
// Mock robots.txt parsing
});
// Test Case 8: File export
it('should export crawled pages as markdown files', async () => {
// Mock file system operations
// Verify file creation
});
// Test Case 9: Webhook notifications
it('should send webhook on completion', async () => {
// Mock HTTP client
// Verify webhook call
});
// Test Case 10: Invalid crawl parameters
it('should validate crawl parameters', async () => {
// Test invalid maxDepth, limit, etc.
});
});
describe('getCrawlStatus', () => {
// Test Case 1: Active crawl status
it('should return status for active crawl', async () => {
// Mock Redis data
// Verify status response
});
// Test Case 2: Completed crawl status
it('should return completed crawl with results', async () => {
// Include exported files info
});
// Test Case 3: Non-existent crawl ID
it('should return 404 for invalid crawl ID', async () => {
// Test missing crawl
});
// Test Case 4: Failed crawl status
it('should include error details for failed crawl', async () => {
// Mock failed job data
});
});
describe('cancelCrawlJob', () => {
// Test Case 1: Cancel active crawl
it('should successfully cancel running crawl', async () => {
// Mock queue cancellation
});
// Test Case 2: Cancel completed crawl
it('should not cancel already completed crawl', async () => {
// Verify appropriate response
});
});
});File: src/api/controllers/__tests__/batch-scrape.controller.test.ts
describe('BatchScrapeController', () => {
describe('initiateBatch', () => {
// Test Case 1: Valid batch creation
it('should create batch job for multiple URLs', async () => {
// Test with array of URLs
});
// Test Case 2: Concurrency limits
it('should respect concurrency parameter', async () => {
// Verify parallel execution limits
});
// Test Case 3: Invalid URLs in batch
it('should handle invalid URLs in batch', async () => {
// Mix of valid and invalid URLs
});
// Test Case 4: Batch size limits
it('should enforce maximum batch size', async () => {
// Test with 100+ URLs
});
// Test Case 5: Webhook configuration
it('should store webhook URL for notifications', async () => {
// Verify webhook storage
});
});
describe('getBatchStatus', () => {
// Test Case 1: Progress tracking
it('should return accurate progress percentage', async () => {
// Mock partial completion
});
// Test Case 2: Individual job statuses
it('should include status for each URL', async () => {
// Test mixed statuses
});
});
describe('downloadResults', () => {
// Test Case 1: ZIP download
it('should generate ZIP archive of results', async () => {
// Mock archiver
});
// Test Case 2: JSON download
it('should provide consolidated JSON', async () => {
// Test JSON format
});
// Test Case 3: Individual file download
it('should download single result by job ID', async () => {
// Test file retrieval
});
});
});File: src/api/controllers/__tests__/map.controller.test.ts
describe('MapController', () => {
describe('discoverUrls', () => {
// Test Case 1: Basic URL discovery
it('should discover URLs from website', async () => {
// Mock discovery service
});
// Test Case 2: Multiple discovery methods
it('should aggregate results from all discovery methods', async () => {
// Test sitemap, crawling, robots.txt
});
// Test Case 3: URL filtering
it('should filter URLs by patterns', async () => {
// Test include/exclude patterns
});
// Test Case 4: Subdomain discovery
it('should discover subdomain URLs when enabled', async () => {
// Test subdomain inclusion
});
// Test Case 5: Discovery limits
it('should respect maxUrls parameter', async () => {
// Test URL count limits
});
// Test Case 6: Cache usage
it('should return cached results when fresh', async () => {
// Mock cache hit
});
// Test Case 7: Timeout handling
it('should timeout long-running discoveries', async () => {
// Test timeout parameter
});
// Test Case 8: Search query filtering
it('should filter URLs by search query', async () => {
// Test search functionality
});
});
describe('getCacheStats', () => {
// Test Case 1: Cache statistics
it('should return cache usage stats', async () => {
// Mock Redis stats
});
});
describe('clearCache', () => {
// Test Case 1: Cache clearing
it('should clear cache for specific URL', async () => {
// Verify cache deletion
});
});
});File: src/scraper/__tests__/scraper-manager.test.ts
describe('ScraperManager', () => {
describe('scrape', () => {
// Test Case 1: Playwright to HTTP fallback
it('should fallback to HTTP when Playwright fails', async () => {
// Mock Playwright failure
// Verify HTTP scraper is called
});
// Test Case 2: Cache integration
it('should cache successful responses', async () => {
// Verify cache write
});
// Test Case 3: Content transformation pipeline
it('should apply transformations in correct order', async () => {
// Test cleaning -> markdown -> LLM
});
// Test Case 4: E-commerce site detection
it('should detect and handle e-commerce sites', async () => {
// Test Amazon, Walmart detection
});
// Test Case 5: Error aggregation
it('should aggregate errors from all stages', async () => {
// Test error collection
});
});
});File: src/services/__tests__/url-discovery.service.test.ts
describe('URLDiscoveryService', () => {
describe('discoverUrls', () => {
// Test Case 1: Sitemap parsing
it('should parse XML sitemaps correctly', async () => {
// Mock sitemap response
});
// Test Case 2: Robots.txt parsing
it('should extract sitemaps from robots.txt', async () => {
// Mock robots.txt content
});
// Test Case 3: Browser crawling
it('should discover URLs via browser crawling', async () => {
// Mock Playwright discovery
});
// Test Case 4: Common paths testing
it('should test common URL paths', async () => {
// Test /api, /docs, etc.
});
// Test Case 5: Rate limiting
it('should apply rate limiting between requests', async () => {
// Verify delays
});
// Test Case 6: Parallel discovery
it('should run discovery methods in parallel', async () => {
// Verify concurrent execution
});
});
describe('streamDiscoverUrls', () => {
// Test Case 1: Streaming URL batches
it('should stream URLs in batches', async () => {
// Test batch handler calls
});
// Test Case 2: Real-time filtering
it('should filter URLs during streaming', async () => {
// Test pattern matching
});
});
});File: src/services/__tests__/browser-pool.service.test.ts
describe('BrowserPoolService', () => {
describe('pool management', () => {
// Test Case 1: Browser instance creation
it('should create browser instances up to pool size', async () => {
// Verify pool initialization
});
// Test Case 2: Browser reuse
it('should reuse browser instances', async () => {
// Track instance usage
});
// Test Case 3: Automatic cleanup
it('should cleanup idle browsers', async () => {
// Test timeout cleanup
});
// Test Case 4: Pool scaling
it('should scale pool based on demand', async () => {
// Test dynamic sizing
});
// Test Case 5: Error recovery
it('should recover from browser crashes', async () => {
// Mock browser crash
});
});
});File: src/services/__tests__/cache.service.test.ts
describe('CacheService', () => {
describe('file-based caching', () => {
// Test Case 1: Cache write
it('should write content to cache file', async () => {
// Mock file system
});
// Test Case 2: Cache read
it('should read cached content', async () => {
// Test cache hit
});
// Test Case 3: TTL expiration
it('should expire cache after TTL', async () => {
// Test time-based expiry
});
// Test Case 4: Cache key generation
it('should generate consistent cache keys', async () => {
// Test key generation
});
// Test Case 5: Compression
it('should compress large cache entries', async () => {
// Test gzip compression
});
});
});File: src/services/__tests__/queue.service.test.ts
describe('QueueService', () => {
describe('job management', () => {
// Test Case 1: Job creation
it('should add jobs to queue', async () => {
// Mock BullMQ
});
// Test Case 2: Job processing
it('should process jobs with worker', async () => {
// Test worker execution
});
// Test Case 3: Job retry
it('should retry failed jobs', async () => {
// Test retry logic
});
// Test Case 4: Priority handling
it('should process high priority jobs first', async () => {
// Test job ordering
});
// Test Case 5: Concurrency control
it('should limit concurrent job processing', async () => {
// Test concurrency limits
});
});
});File: src/scraper/__tests__/playwright-scraper.test.ts
describe('PlaywrightScraper', () => {
describe('scrape', () => {
// Test Case 1: Basic page scraping
it('should scrape page content', async () => {
// Mock page content
});
// Test Case 2: JavaScript rendering
it('should wait for JavaScript content', async () => {
// Test dynamic content
});
// Test Case 3: Selector waiting
it('should wait for specific selectors', async () => {
// Test waitForSelector
});
// Test Case 4: Browser actions
it('should execute browser actions', async () => {
// Test click, scroll, fill
});
// Test Case 5: Screenshot capture
it('should capture full page screenshots', async () => {
// Test screenshot
});
// Test Case 6: Resource blocking
it('should block ads and trackers', async () => {
// Test resource interception
});
// Test Case 7: Stealth mode
it('should evade bot detection', async () => {
// Test stealth plugins
});
// Test Case 8: Proxy usage
it('should route through proxy', async () => {
// Test proxy configuration
});
// Test Case 9: Cookie handling
it('should set custom cookies', async () => {
// Test cookie injection
});
// Test Case 10: Timeout management
it('should respect timeout settings', async () => {
// Test page timeout
});
});
});File: src/scraper/__tests__/http-scraper.test.ts
describe('HttpScraper', () => {
describe('scrape', () => {
// Test Case 1: Basic HTTP request
it('should fetch page via HTTP', async () => {
// Mock axios response
});
// Test Case 2: Custom headers
it('should include custom headers', async () => {
// Test header injection
});
// Test Case 3: User agent rotation
it('should use custom user agent', async () => {
// Test UA string
});
// Test Case 4: Error handling
it('should handle HTTP errors', async () => {
// Test 404, 500, etc.
});
// Test Case 5: Redirect following
it('should follow redirects', async () => {
// Test 301/302 redirects
});
});
});File: src/transformers/__tests__/html-to-markdown.test.ts
describe('HTMLToMarkdown', () => {
describe('convert', () => {
// Test Case 1: Basic HTML elements
it('should convert basic HTML to markdown', async () => {
// Test p, h1-h6, lists, etc.
});
// Test Case 2: Link preservation
it('should preserve links with titles', async () => {
// Test anchor tags
});
// Test Case 3: Image handling
it('should convert images with alt text', async () => {
// Test img tags
});
// Test Case 4: Table conversion
it('should convert HTML tables', async () => {
// Test table structure
});
// Test Case 5: Code blocks
it('should preserve code formatting', async () => {
// Test pre/code tags
});
// Test Case 6: Nested structures
it('should handle nested elements', async () => {
// Test complex nesting
});
// Test Case 7: Special characters
it('should escape markdown characters', async () => {
// Test *, _, `, etc.
});
// Test Case 8: Whitespace handling
it('should normalize whitespace', async () => {
// Test spacing/newlines
});
});
});File: src/transformers/__tests__/content-cleaner.test.ts
describe('ContentCleaner', () => {
describe('clean', () => {
// Test Case 1: Script removal
it('should remove script tags', async () => {
// Test JavaScript removal
});
// Test Case 2: Style removal
it('should remove style tags and attributes', async () => {
// Test CSS removal
});
// Test Case 3: Comment removal
it('should remove HTML comments', async () => {
// Test comment stripping
});
// Test Case 4: Ad element removal
it('should remove ad containers', async () => {
// Test ad detection
});
// Test Case 5: Navigation removal
it('should remove nav elements', async () => {
// Test header/footer removal
});
// Test Case 6: Main content extraction
it('should identify main content', async () => {
// Test content detection
});
});
});File: src/transformers/__tests__/llm-extractor.test.ts
describe('LLMExtractor', () => {
describe('extract', () => {
// Test Case 1: Schema extraction
it('should extract data matching schema', async () => {
// Mock OpenAI response
});
// Test Case 2: Array extraction
it('should extract arrays of items', async () => {
// Test list extraction
});
// Test Case 3: Nested object extraction
it('should handle nested schemas', async () => {
// Test complex schemas
});
// Test Case 4: Token limit handling
it('should truncate content for token limits', async () => {
// Test content truncation
});
// Test Case 5: Error recovery
it('should handle LLM errors gracefully', async () => {
// Test API failures
});
// Test Case 6: Response validation
it('should validate extracted data', async () => {
// Test schema validation
});
});
});File: src/utils/__tests__/url-validation.utils.test.ts
describe('URLValidationUtils', () => {
describe('isValidUrl', () => {
// Test Case 1: Valid URLs
it('should validate correct URLs', () => {
// Test http, https, with/without www
});
// Test Case 2: Invalid URLs
it('should reject invalid URLs', () => {
// Test malformed URLs
});
// Test Case 3: Special protocols
it('should handle special protocols', () => {
// Test ftp, file, javascript
});
});
describe('normalizeUrl', () => {
// Test Case 1: URL normalization
it('should normalize URLs consistently', () => {
// Test trailing slashes, fragments
});
// Test Case 2: Query parameter handling
it('should sort query parameters', () => {
// Test parameter ordering
});
});
describe('filterAndSortUrls', () => {
// Test Case 1: Pattern matching
it('should filter by include/exclude patterns', () => {
// Test regex patterns
});
// Test Case 2: Domain filtering
it('should filter by domain', () => {
// Test subdomain handling
});
// Test Case 3: Deduplication
it('should remove duplicate URLs', () => {
// Test URL uniqueness
});
});
});File: src/__tests__/integration/scraping/scraping.integration.test.ts
describe('Scraping Integration', () => {
// Test Case 1: Full scraping pipeline
it('should scrape, transform, and extract data', async () => {
// Test complete flow
});
// Test Case 2: Cache integration
it('should use cache across requests', async () => {
// Test cache hit/miss scenarios
});
// Test Case 3: Queue processing
it('should process scraping through queue', async () => {
// Test async job processing
});
// Test Case 4: Error handling pipeline
it('should handle errors at each stage', async () => {
// Test error propagation
});
});File: src/__tests__/integration/crawling/crawling.integration.test.ts
describe('Crawling Integration', () => {
// Test Case 1: Multi-page crawling
it('should crawl multiple pages with depth control', async () => {
// Test URL discovery and scraping
});
// Test Case 2: File export
it('should export crawled pages to files', async () => {
// Test markdown file generation
});
// Test Case 3: Streaming crawl
it('should stream URLs during discovery', async () => {
// Test real-time processing
});
// Test Case 4: Webhook notifications
it('should send webhooks on events', async () => {
// Test webhook delivery
});
});File: src/__tests__/e2e/api-workflows.e2e.test.ts
describe('E2E API Workflows', () => {
// Test Case 1: Scrape endpoint
it('should scrape URL via API', async () => {
// Full API request/response
});
// Test Case 2: Crawl workflow
it('should start and monitor crawl', async () => {
// Test crawl lifecycle
});
// Test Case 3: Batch processing
it('should process batch and download results', async () => {
// Test batch workflow
});
// Test Case 4: Map and scrape
it('should discover and scrape URLs', async () => {
// Test combined workflow
});
});// Mock Redis
export const mockRedis = {
get: jest.fn(),
set: jest.fn(),
del: jest.fn(),
keys: jest.fn(),
exists: jest.fn()
};
// Mock Playwright
export const mockPage = {
goto: jest.fn(),
content: jest.fn(),
evaluate: jest.fn(),
waitForSelector: jest.fn(),
screenshot: jest.fn(),
close: jest.fn()
};
// Mock OpenAI
export const mockOpenAI = {
chat: {
completions: {
create: jest.fn()
}
}
};
// Mock BullMQ
export const mockQueue = {
add: jest.fn(),
process: jest.fn(),
on: jest.fn(),
close: jest.fn()
};// HTML fixtures
export const fixtures = {
simple: '<html><body><h1>Test</h1><p>Content</p></body></html>',
complex: fs.readFileSync('__tests__/fixtures/complex.html', 'utf-8'),
ecommerce: fs.readFileSync('__tests__/fixtures/amazon.html', 'utf-8')
};
// Response fixtures
export const responses = {
success: { status: 200, data: { content: 'test' } },
error: { status: 500, error: 'Internal error' }
};Add to package.json:
{
"scripts": {
"test": "jest",
"test:watch": "jest --watch",
"test:coverage": "jest --coverage",
"test:unit": "jest --testPathPattern=unit",
"test:integration": "jest --testPathPattern=integration",
"test:e2e": "jest --testPathPattern=e2e",
"test:ci": "jest --ci --coverage --maxWorkers=2"
}
}# .github/workflows/test.yml
name: Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- uses: actions/setup-node@v2
- run: npm ci
- run: npm run test:ci
- uses: codecov/codecov-action@v1- Statements: 80%
- Branches: 80%
- Functions: 80%
- Lines: 80%
- API Controllers: 90%+
- Core Services: 85%+
- Scrapers: 85%+
- Transformers: 90%+
- Type definitions
- Index files
- Configuration files
- Generated code
-
Arrange-Act-Assert Pattern
- Clear test structure
- Single assertion per test
- Descriptive test names
-
Mock External Dependencies
- Never make real API calls
- Mock file system operations
- Isolate unit tests
-
Use Test Builders
- Create reusable test data
- Reduce test setup code
- Maintain consistency
-
Async Testing
- Always await async operations
- Use proper error handling
- Test both success and failure
-
Performance Testing
- Monitor test execution time
- Mock slow operations
- Use timeouts appropriately
- Jest setup and configuration
- Core scraper tests (ScraperManager, PlaywrightScraper)
- Critical transformer tests (HTMLToMarkdown, LLMExtractor)
- Controller tests (all endpoints)
- Service tests (cache, queue, URL discovery)
- Integration tests for main workflows
- Remaining utility tests
- E2E test scenarios
- Performance test suite
- CI/CD integration
-
Update tests when code changes
- Keep tests in sync with implementation
- Add tests for new features
- Refactor tests when needed
-
Monitor test coverage
- Run coverage reports regularly
- Address coverage gaps
- Focus on critical paths
-
Review test failures
- Fix flaky tests immediately
- Investigate root causes
- Update mocks as needed
-
Performance monitoring
- Track test suite execution time
- Optimize slow tests
- Parallelize when possible
This comprehensive test suite will ensure DeepScraper's reliability, maintainability, and quality. The tests cover all critical paths, handle edge cases, and provide confidence for refactoring and new feature development. Implementation should follow the priority phases to quickly achieve meaningful coverage while building towards comprehensive testing.