93 lines
4.7 KiB
Markdown
93 lines
4.7 KiB
Markdown
# Elasticsearch Recon Ingestion Scripts (ERIS)
|
|
> A utility for ingesting various large scale reconnaissance data logs into Elasticsearch
|
|
|
|
## Prerequisites
|
|
- [python](https://www.python.org/)
|
|
- [elasticsearch](https://pypi.org/project/elasticsearch/) *(`pip install elasticsearch`)*
|
|
- [aiofiles](https://pypi.org/project/aiofiles) *(`pip install aiofiles`)*
|
|
- [aiohttp](https://pypi.org/projects/aiohttp) *(`pip install aiohttp`)*
|
|
|
|
## Usage
|
|
```shell
|
|
python eris.py [options] <input>
|
|
```
|
|
**Note:** The `<input>` can be a file or a directory of files, depending on the ingestion script.
|
|
|
|
### Options
|
|
###### General arguments
|
|
| Argument | Description |
|
|
|--------------|-----------------------------------------------|
|
|
| `input_path` | Path to the input file or directory |
|
|
| `--watch` | Create or watch a FIFO for real-time indexing |
|
|
|
|
###### Elasticsearch arguments
|
|
| Argument | Description | Default |
|
|
|-----------------|---------------------------------------------------------|---------------------|
|
|
| `--host` | Elasticsearch host | `http://localhost/` |
|
|
| `--port` | Elasticsearch port | `9200` |
|
|
| `--user` | Elasticsearch username | `elastic` |
|
|
| `--password` | Elasticsearch password | `$ES_PASSWORD` |
|
|
| `--api-key` | Elasticsearch API Key for authentication | `$ES_APIKEY` |
|
|
| `--self-signed` | Elasticsearch connection with a self-signed certificate | |
|
|
|
|
###### Elasticsearch indexing arguments
|
|
| Argument | Description | Default |
|
|
|--------------|--------------------------------------|---------------------|
|
|
| `--index` | Elasticsearch index name | Depends on ingestor |
|
|
| `--pipeline` | Use an ingest pipeline for the index | |
|
|
| `--replicas` | Number of replicas for the index | `1` |
|
|
| `--shards` | Number of shards for the index | `1` |
|
|
|
|
###### Performance arguments
|
|
| Argument | Description | Default |
|
|
|-------------------|----------------------------------------------------------|---------|
|
|
| `--chunk-max` | Maximum size in MB of a chunk | `100` |
|
|
| `--chunk-size` | Number of records to index in a chunk | `50000` |
|
|
| `--retries` | Number of times to retry indexing a chunk before failing | `100` |
|
|
| `--timeout` | Number of seconds to wait before retrying a chunk | `60` |
|
|
|
|
###### Ingestion arguments
|
|
| Argument | Description |
|
|
|-------------|--------------------------|
|
|
| `--certs` | Index Certstream records |
|
|
| `--httpx` | Index HTTPX records |
|
|
| `--masscan` | Index Masscan records |
|
|
| `--massdns` | Index massdns records |
|
|
| `--zone` | Index zone DNS records |
|
|
|
|
This ingestion suite will use the built in node sniffer, so by connecting to a single node, you can load balance across the entire cluster.
|
|
It is good to know how much nodes you have in the cluster to determine how to fine tune the arguments for the best performance, based on your environment.
|
|
|
|
## GeoIP Pipeline
|
|
Create & add a geoip pipeline and use the following in your index mappings:
|
|
|
|
```json
|
|
"geoip": {
|
|
"city_name": "City",
|
|
"continent_name": "Continent",
|
|
"country_iso_code": "CC",
|
|
"country_name": "Country",
|
|
"location": {
|
|
"lat": 0.0000,
|
|
"lon": 0.0000
|
|
},
|
|
"region_iso_code": "RR",
|
|
"region_name": "Region"
|
|
}
|
|
```
|
|
|
|
## Changelog
|
|
- Added ingestion script for certificate transparency logs in real time using websockets.
|
|
- `--dry-run` removed as this nears production level
|
|
- Implemented [async elasticsearch](https://elasticsearch-py.readthedocs.io/en/latest/async.html) into the codebase & refactored some of the logic to accomadate.
|
|
- The `--watch` feature now uses a FIFO to do live ingestion.
|
|
- Isolated eris.py into it's own file and seperated the ingestion agents into their own modules.
|
|
|
|
## Roadmap
|
|
- Fix issue with `ingest_certs.py` and not needing to pass a file to it
|
|
- WHOIS database ingestion scripts
|
|
- Dynamically update the batch metrics when the sniffer adds or removes nodes
|
|
|
|
___
|
|
|
|
###### Mirrors for this repository: [acid.vegas](https://git.acid.vegas/eris) • [SuperNETs](https://git.supernets.org/acidvegas/eris) • [GitHub](https://github.com/acidvegas/eris) • [GitLab](https://gitlab.com/acidvegas/eris) • [Codeberg](https://codeberg.org/acidvegas/eris) |