143 lines
3.8 KiB
Markdown
143 lines
3.8 KiB
Markdown
# PyLCG
|
|
> Ultra-fast Linear Congruential Generator for IP Sharding
|
|
|
|
PyLCG is a high-performance Python implementation of a memory-efficient IP address sharding system using Linear Congruential Generators (LCG) for deterministic random number generation. This tool enables distributed scanning & network reconnaissance by efficiently dividing IP ranges across multiple machines while maintaining pseudo-random ordering.
|
|
|
|
## Features
|
|
|
|
- Memory-efficient IP range processing
|
|
- Deterministic pseudo-random IP generation
|
|
- High-performance LCG implementation
|
|
- Support for sharding across multiple machines
|
|
- Zero dependencies beyond Python standard library
|
|
- Simple command-line interface
|
|
|
|
## Installation
|
|
|
|
### From PyPI
|
|
```bash
|
|
pip install pylcg
|
|
```
|
|
|
|
### From Source
|
|
```bash
|
|
git clone https://github.com/acidvegas/pylcg
|
|
cd pylcg
|
|
chmod +x pylcg.py
|
|
```
|
|
|
|
## Usage
|
|
|
|
### Command Line
|
|
|
|
```bash
|
|
./pylcg.py 192.168.0.0/16 --shard-num 1 --total-shards 4 --seed 12345
|
|
```
|
|
|
|
### As a Library
|
|
|
|
```python
|
|
from pylcg import ip_stream
|
|
|
|
# Generate IPs for the first shard of 4 total shards
|
|
for ip in ip_stream('192.168.0.0/16', shard_num=1, total_shards=4, seed=12345):
|
|
print(ip)
|
|
```
|
|
|
|
## How It Works
|
|
|
|
### Linear Congruential Generator
|
|
|
|
PyLCG uses an optimized LCG implementation with carefully chosen parameters:
|
|
| Name | Variable | Value |
|
|
|------------|----------|--------------|
|
|
| Multiplier | `a` | `1664525` |
|
|
| Increment | `c` | `1013904223` |
|
|
| Modulus | `m` | `2^32` |
|
|
|
|
This generates a deterministic sequence of pseudo-random numbers using the formula:
|
|
```
|
|
next = (a * current + c) mod m
|
|
```
|
|
|
|
### Memory-Efficient IP Processing
|
|
|
|
Instead of loading entire IP ranges into memory, PyLCG:
|
|
1. Converts CIDR ranges to start/end integers
|
|
2. Uses generator functions for lazy evaluation
|
|
3. Calculates IPs on-demand using index mapping
|
|
4. Maintains constant memory usage regardless of range size
|
|
|
|
### Sharding Algorithm
|
|
|
|
The sharding system uses an interleaved approach:
|
|
1. Each shard is assigned a subset of indices based on modulo arithmetic
|
|
2. The LCG randomizes the order within each shard
|
|
3. Work is distributed evenly across shards
|
|
4. No sequential scanning patterns
|
|
|
|
## Performance
|
|
|
|
PyLCG is designed for maximum performance:
|
|
- Generates millions of IPs per second
|
|
- Constant memory usage (~100KB)
|
|
- Minimal CPU overhead
|
|
- No disk I/O required
|
|
|
|
Benchmark results on a typical system:
|
|
- IP Generation: ~5-10 million IPs/second
|
|
- Memory Usage: < 1MB for any range size
|
|
- LCG Operations: < 1 microsecond per number
|
|
|
|
## Contributing
|
|
|
|
### Performance Optimization
|
|
|
|
We welcome contributions that improve PyLCG's performance. When submitting optimizations:
|
|
|
|
1. Run the included benchmark suite:
|
|
```bash
|
|
python3 unit_test.py
|
|
```
|
|
|
|
2. Include before/after benchmark results for:
|
|
- IP generation speed
|
|
- Memory usage
|
|
- LCG sequence generation
|
|
- Shard distribution metrics
|
|
|
|
3. Consider optimizing:
|
|
- Number generation algorithms
|
|
- Memory access patterns
|
|
- CPU cache utilization
|
|
- Python-specific optimizations
|
|
|
|
4. Document any tradeoffs between:
|
|
- Speed vs memory usage
|
|
- Randomness vs performance
|
|
- Complexity vs maintainability
|
|
|
|
### Benchmark Guidelines
|
|
|
|
When running benchmarks:
|
|
1. Use consistent hardware/environment
|
|
2. Run multiple iterations
|
|
3. Test with various CIDR ranges
|
|
4. Measure both average and worst-case performance
|
|
5. Profile memory usage patterns
|
|
6. Test shard distribution uniformity
|
|
|
|
## Roadmap
|
|
|
|
- [ ] IPv6 support
|
|
- [ ] Custom LCG parameters
|
|
- [ ] Configurable chunk sizes
|
|
- [ ] State persistence
|
|
- [ ] Resume capability
|
|
- [ ] S3/URL input support
|
|
- [ ] Extended benchmark suite
|
|
|
|
---
|
|
|
|
###### Mirrors: [acid.vegas](https://git.acid.vegas/pylcg) • [SuperNETs](https://git.supernets.org/acidvegas/pylcg) • [GitHub](https://github.com/acidvegas/pylcg) • [GitLab](https://gitlab.com/acidvegas/pylcg) • [Codeberg](https://codeberg.org/acidvegas/pylcg)
|