eris/README.md

50 lines
3.6 KiB
Markdown
Raw Normal View History

2024-01-20 07:04:50 +00:00
# Elasticsearch Recon Ingestion Scripts (ERIS)
> A utility for ingesting various large scale reconnaissance data logs into Elasticsearch
2024-01-20 07:04:50 +00:00
### Work In Progress
## Prerequisites
- [python](https://www.python.org/)
- [elasticsearch](https://pypi.org/project/elasticsearch/) *(`pip install elasticsearch`)*
## Usage
```shell
python ingest_XXXX.py [options] <input>
```
**Note:** The `<input>` can be a file or a directory of files, depending on the ingestion script.
2024-01-20 07:04:50 +00:00
###
###### General arguments
| Argument | Description |
|-------------------|----------------------------------------------------------------|
| `input_path` | Path to the input file or directory |
| `--dry-run` | Dry run *(do not index records to Elasticsearch)* |
| `--watch` | Watch the input file for new lines and index them in real time |
###### Elasticsearch arguments
| Argument | Description | Default |
|-------------------|--------------------------------------------------------------------------------------|---------------|
| `--host` | Elasticsearch host *(Will sniff for other nodes in the cluster)* | `localhost` |
| `--port` | Elasticsearch port | `9200` |
| `--user` | Elasticsearch username | `elastic` |
| `--password` | Elasticsearch password *(if not provided, check environment variable `ES_PASSWORD`)* | |
| `--api-key` | Elasticsearch API Key for authentication | |
| `--self-signed` | Elastic search instance is using a self-signed certificate | `true` |
| `--index` | Elasticsearch index name | `masscan-logs`|
| `--shards` | Number of shards for the index | `1` |
| `--replicas` | Number of replicas for the index | `1` |
###### Performance arguments
| Argument | Description | Default |
|-------------------|--------------------------------------------------------------------------------------|---------------|
| `--batch-max` | Maximum size in MB of a batch | `10` |
| `--batch-size` | Number of records to index in a batch | `5000` |
| `--batch-threads` | Number of threads to use when indexing in batches | `2` |
| `--retries` | Number of times to retry indexing a batch before failing | `10` |
| `--timeout` | Number of seconds to wait before retrying a batch | `30` |
**NOTE:** Using `--batch-threads` as 4 and `--batch-size` as 10000 with 3 nodes would process 120,000 records before indexing 40,000 per node.
2024-01-20 07:04:50 +00:00
___
2024-01-21 02:37:27 +00:00
###### Mirrors for this repository: [acid.vegas](https://git.acid.vegas/eris) • [SuperNETs](https://git.supernets.org/acidvegas/eris) • [GitHub](https://github.com/acidvegas/eris) • [GitLab](https://gitlab.com/acidvegas/eris) • [Codeberg](https://codeberg.org/acidvegas/eris)