Web Spider
A distributed web crawler in Rust with a Redis-backed BFS queue, URL normalisation and deduplication, and configurable concurrency and depth, orchestrated with Docker Compose.
- Role
- Solo Developer
- Status
- Completed
A distributed web crawler in Rust with a Redis-backed BFS queue, URL normalisation and deduplication, and configurable concurrency and depth, orchestrated with Docker Compose.