Compare commits

...

2 Commits
v1.0.1 ... main

View File

@ -40,19 +40,21 @@ Where:
- `TOTAL` is the total number of shards
### Examples
Let's say you have a very large list of domains and you want to do recon on each domain. Using a single machine, this could take a very long time. However, you can split the workload across multiple machines:
- Machine number 1 would run:
```bash
curl https://example.com/large_file.txt | shardz 1/3
curl https://example.com/datasets/large_domain_list.txt | shardz 1/3 | httpx -title -ip -tech-detect -json -o shard-1.json
```
- Machine number 2 would run:
```bash
curl https://example.com/large_file.txt | shardz 2/3
curl https://example.com/datasets/large_domain_list.txt | shardz 2/3 | httpx -title -ip -tech-detect -json -o shard-2.json
```
- Machine number 3 would run:
```bash
curl https://example.com/large_file.txt | shardz 3/3
curl https://example.com/datasets/large_domain_list.txt | shardz 3/3 | httpx -title -ip -tech-detect -json -o shard-3.json
```
## How It Works
@ -64,6 +66,23 @@ Shardz uses a modulo operation to determine which lines should be processed by e
This ensures an even distribution of the workload across all shards.
## Simplicity
For what its worth, the same functionality of this tool can be done with a bash function in your `.bashrc`:
```bash
shardz() {
awk -v n="$1" -v t="$2" 'NR % t == n'
}
```
```bash
cat domains.txt | shardz 1/3 | httpx -title -ip -tech-detect -json -o shard-1.json
cat domains.txt | shardz 2/3 | httpx -title -ip -tech-detect -json -o shard-2.json
cat domains.txt | shardz 3/3 | httpx -title -ip -tech-detect -json -o shard-3.json
```
This was just a fun little project to brush up on my C, and to explore the requirements to having a package added to Linux package manager repositories.
---
###### Mirrors: [acid.vegas](https://git.acid.vegas/shardz) • [SuperNETs](https://git.supernets.org/acidvegas/shardz) • [GitHub](https://github.com/acidvegas/shardz) • [GitLab](https://gitlab.com/acidvegas/shardz) • [Codeberg](https://codeberg.org/acidvegas/shardz)