18 Commits

Author	SHA1	Message	Date
Dionysus	603d005a47	Added IXP ingestor, updated all other ingestors	2024-03-19 19:00:12 -04:00
Dionysus	87f2cf27ea	Code cleanup	2024-03-11 22:33:18 -04:00
Dionysus	b018da4e4d	Full source commenting, uniformity in testing function, records stored as a list by default incase an IP address yields multiple PTR records	2024-03-11 19:18:03 -04:00
Dionysus	6c4ae3e988	Many bugs fixed in sniffer and async model.	2024-03-08 12:13:57 -05:00
Dionysus	d34aa105f1	Overall code cleanup	2024-03-08 00:07:26 -05:00
Dionysus	45f878285c	Testing function added to every ingestor to debug directly. No more --dry-run needed.	2024-03-07 23:31:30 -05:00
Dionysus	b78b99e060	Changed yield to return when sentinal value is recieved through the FIFO	2024-03-07 22:57:59 -05:00
Dionysus	da065c6889	Added a testing function for calling the ingest_massdns.py file directly for debugging	2024-03-07 22:55:48 -05:00
Dionysus	9c6beb71ce	MassDNS ingestion script now caches the previous record to support IP addresses that yield more than one PTR record (field turned into a list when +1). Records will now upsert so MassDNS can be streaming into ES 24/7=	2024-03-07 21:57:10 -05:00
Dionysus	b15b3d8241	OCD about formatting again	2024-03-06 15:07:52 -05:00
Dionysus	90d6260b99	Updated massdns ingestion script with sentinal value checking and using the ip address as the document id	2024-03-06 14:33:21 -05:00
Dionysus	a4b89e6e5a	Asyncronous refactorization pushed as main version 💯	2024-03-05 22:19:11 -05:00
Dionysus	c05c48f3fe	Started asyncronous implementation of bulk streaming data, altered ERIS defaults, etc	2024-03-04 17:44:09 -05:00
Dionysus	1ee71868ec	Ingestion agents are now modular, FIFO live ingestion added	2024-02-02 00:11:18 -05:00
Dionysus	c105db705d	Updated README, copied over consistencies across the ingestors, docstring updates to reflect on new arguments	2024-01-27 04:28:30 -05:00
Dionysus	c481edc883	Added parallel bulk uploading, error handling, sniffing nodes for discovery, dynamic batch sizes, and more	2024-01-27 01:13:11 -05:00
Dionysus	3ff233a991	Updated README, fixed issue using the wrong domain in records for zone file ingestion (woops)	2024-01-20 10:53:55 -05:00
Dionysus	14b6d1c88a	Initial commit	2024-01-20 02:04:50 -05:00