Compare commits
2 Commits
Author | SHA1 | Date | |
---|---|---|---|
e0228615dc | |||
778fb772ec |
25
README.md
25
README.md
@ -40,19 +40,21 @@ Where:
|
||||
- `TOTAL` is the total number of shards
|
||||
|
||||
### Examples
|
||||
Let's say you have a very large list of domains and you want to do recon on each domain. Using a single machine, this could take a very long time. However, you can split the workload across multiple machines:
|
||||
|
||||
- Machine number 1 would run:
|
||||
```bash
|
||||
curl https://example.com/large_file.txt | shardz 1/3
|
||||
curl https://example.com/datasets/large_domain_list.txt | shardz 1/3 | httpx -title -ip -tech-detect -json -o shard-1.json
|
||||
```
|
||||
|
||||
- Machine number 2 would run:
|
||||
```bash
|
||||
curl https://example.com/large_file.txt | shardz 2/3
|
||||
curl https://example.com/datasets/large_domain_list.txt | shardz 2/3 | httpx -title -ip -tech-detect -json -o shard-2.json
|
||||
```
|
||||
|
||||
- Machine number 3 would run:
|
||||
```bash
|
||||
curl https://example.com/large_file.txt | shardz 3/3
|
||||
curl https://example.com/datasets/large_domain_list.txt | shardz 3/3 | httpx -title -ip -tech-detect -json -o shard-3.json
|
||||
```
|
||||
|
||||
## How It Works
|
||||
@ -64,6 +66,23 @@ Shardz uses a modulo operation to determine which lines should be processed by e
|
||||
|
||||
This ensures an even distribution of the workload across all shards.
|
||||
|
||||
## Simplicity
|
||||
|
||||
For what its worth, the same functionality of this tool can be done with a bash function in your `.bashrc`:
|
||||
```bash
|
||||
shardz() {
|
||||
awk -v n="$1" -v t="$2" 'NR % t == n'
|
||||
}
|
||||
```
|
||||
|
||||
```bash
|
||||
cat domains.txt | shardz 1/3 | httpx -title -ip -tech-detect -json -o shard-1.json
|
||||
cat domains.txt | shardz 2/3 | httpx -title -ip -tech-detect -json -o shard-2.json
|
||||
cat domains.txt | shardz 3/3 | httpx -title -ip -tech-detect -json -o shard-3.json
|
||||
```
|
||||
|
||||
This was just a fun little project to brush up on my C, and to explore the requirements to having a package added to Linux package manager repositories.
|
||||
|
||||
---
|
||||
|
||||
###### Mirrors: [acid.vegas](https://git.acid.vegas/shardz) • [SuperNETs](https://git.supernets.org/acidvegas/shardz) • [GitHub](https://github.com/acidvegas/shardz) • [GitLab](https://gitlab.com/acidvegas/shardz) • [Codeberg](https://codeberg.org/acidvegas/shardz)
|
||||
|
Loading…
Reference in New Issue
Block a user