Commit Graph

20 Commits

Author SHA1 Message Date
d34aa105f1
Overall code cleanup 2024-03-08 00:07:26 -05:00
45f878285c
Testing function added to every ingestor to debug directly. No more --dry-run needed. 2024-03-07 23:31:30 -05:00
b78b99e060
Changed yield to return when sentinal value is recieved through the FIFO 2024-03-07 22:57:59 -05:00
da065c6889
Added a testing function for calling the ingest_massdns.py file directly for debugging 2024-03-07 22:55:48 -05:00
9c6beb71ce
MassDNS ingestion script now caches the previous record to support IP addresses that yield more than one PTR record (field turned into a list when +1). Records will now upsert so MassDNS can be streaming into ES 24/7= 2024-03-07 21:57:10 -05:00
b15b3d8241
OCD about formatting again 2024-03-06 15:07:52 -05:00
fd617e3c9d
Certstream ingestion improved, still need to test stripping nulls and refactor the index mapping so we only store whats needed 2024-03-06 14:38:34 -05:00
90d6260b99
Updated massdns ingestion script with sentinal value checking and using the ip address as the document id 2024-03-06 14:33:21 -05:00
cba51ca2dd
Added a note about NSEC3 hash cracking for zone file data 2024-03-06 14:16:05 -05:00
654e4a8667
Zone file ingestion script now uses the same sentinal value as masscans ingestion, set document id as the domain name to allow updating records if they exist 2024-03-06 14:12:27 -05:00
84f124b23d
Masscan ingestion script updated to use ip:port as the document id to allow updating records that already exist. Added a sentinal value to trigger an EOF when using --watch with FIFO's 2024-03-06 13:26:45 -05:00
a4b89e6e5a
Asyncronous refactorization pushed as main version 💯 2024-03-05 22:19:11 -05:00
c05c48f3fe
Started asyncronous implementation of bulk streaming data, altered ERIS defaults, etc 2024-03-04 17:44:09 -05:00
1ee71868ec
Ingestion agents are now modular, FIFO live ingestion added 2024-02-02 00:11:18 -05:00
c105db705d
Updated README, copied over consistencies across the ingestors, docstring updates to reflect on new arguments 2024-01-27 04:28:30 -05:00
88e0dbfea8
Added parallel bulk uploading, error handling, sniffing nodes for discovery, dynamic batch sizes, and more 2024-01-27 01:13:27 -05:00
c481edc883
Added parallel bulk uploading, error handling, sniffing nodes for discovery, dynamic batch sizes, and more 2024-01-27 01:13:11 -05:00
b14038a335
Updated mirrors 2024-01-20 21:37:27 -05:00
3ff233a991
Updated README, fixed issue using the wrong domain in records for zone file ingestion (woops) 2024-01-20 10:53:55 -05:00
14b6d1c88a
Initial commit 2024-01-20 02:04:50 -05:00