acidvegas/eris
acidvegas
/
eris
1
Fork 0
Code Issues Pull Requests Activity
57 Commits 1 Branch 0 Tags 395 KiB
Commit Graph

57 Commits

Author SHA1 Message Date
Dionysus 124e4b0cf3
ECS formatting added to eris, certstream ingestor now caches to prevent duplication, elastic connections properly closed now 2024-03-23 22:47:30 -04:00
Dionysus 510f7db07e
Fixed _doc to doc in record 2024-03-23 17:26:55 -04:00
Dionysus c6204be8e1
Added more comments and thoughts 2024-03-23 15:31:15 -04:00
Dionysus 20ee3fcb0f
Added a comment about the methodology of determining the zone the data is from. 2024-03-23 14:48:45 -04:00
Dionysus e74741c0ad
Updated example record comment under source to reflect on the new record structure 2024-03-23 14:46:44 -04:00
Dionysus 6983b8ebec
AXFR detection in zone file input, to determine if the data is form ICANN or an AXFR against a dns (for source tracking) 2024-03-23 14:42:51 -04:00
Dionysus a53541c328
Added zone field too better querying on specific tlds/zones 2024-03-23 14:31:39 -04:00
Dionysus 78d7556f09
Records stored as a nested type for better querying, added source field so we can identify where the zone data derived (icann, axfr, breach, etc) 2024-03-23 13:46:38 -04:00
Dionysus 603d005a47
Added IXP ingestor, updated all other ingestors 2024-03-19 19:00:12 -04:00
Dionysus 24850ea976
Typo in pypi title 2024-03-15 01:26:55 -04:00
Dionysus fe49255f69
Added elastic common schema (ecs) logging to file for ingesting eris logs straight into ES 2024-03-15 01:25:09 -04:00
Dionysus 1ab7199f7d
Certstream ingestor now only logs sub-domains since we already ingested zone files. Ignores www. and wildcard domains. 2024-03-13 22:34:20 -04:00
Dionysus 7f93a4d8de
add ingest_rir_transfers.py 2024-03-13 20:54:36 -04:00
Dionysus 4dc31a5090
RIR Transfers ingestor added 2024-03-13 20:51:53 -04:00
Dionysus bd735ea8a7
Thats how you fucking do it. Delegations stored in ranges now for querying. 2024-03-12 21:20:34 -04:00
Dionysus 1864f08e4b
Fixed import error saying aiofiles instead of aiohttp 2024-03-12 18:23:52 -04:00
Dionysus b1fa34f3aa
Added anomaly detection to RIR delegations ingestor 2024-03-12 18:19:47 -04:00
Dionysus 00711fe856
Created an ingestor for RIR delegations 2024-03-12 17:04:14 -04:00
Dionysus 6bb0d4eeef
Lowered reconnect time on certstream timeout 2024-03-12 00:44:17 -04:00
Dionysus 67c7e639a9
Renamed default_index to eris-certstream 2024-03-11 23:32:37 -04:00
Dionysus 1ad7d8a1d3
Fixed logger again 2024-03-11 23:30:48 -04:00
Dionysus 32d5c773e8
Fixed missing logging.handlers import 2024-03-11 23:07:35 -04:00
Dionysus c3b9cb9e08
Fixed logger level 2024-03-11 23:06:25 -04:00
Dionysus de3878ef6b
Removed _id from certstream yield and renamed ingest_certs.py to ingest_certstream.py 2024-03-11 22:46:48 -04:00
Dionysus 87f2cf27ea
Code cleanup 2024-03-11 22:33:18 -04:00
Dionysus b018da4e4d
Full source commenting, uniformity in testing function, records stored as a list by default incase an IP address yields multiple PTR records 2024-03-11 19:18:03 -04:00
Dionysus 6c4ae3e988
Many bugs fixed in sniffer and async model. 2024-03-08 12:13:57 -05:00
Dionysus d34aa105f1
Overall code cleanup 2024-03-08 00:07:26 -05:00
Dionysus 681e3bd788
Fixed issue with ingest_certs and the ingestion function signature. Simple placeholder argument (un-used) added to maintain function uniformity 2024-03-07 23:33:20 -05:00
Dionysus 45f878285c
Testing function added to every ingestor to debug directly. No more --dry-run needed. 2024-03-07 23:31:30 -05:00
Dionysus b78b99e060
Changed yield to return when sentinal value is recieved through the FIFO 2024-03-07 22:57:59 -05:00
Dionysus da065c6889
Added a testing function for calling the ingest_massdns.py file directly for debugging 2024-03-07 22:55:48 -05:00
Dionysus 3c98f8f1f1
MassDNS ingestion script now caches the previous record to support IP addresses that yield more than one PTR record (field turned into a list when +1). Records will now upsert so MassDNS can be streaming into ES 24/7= 2024-03-07 21:57:44 -05:00
Dionysus 9c6beb71ce
MassDNS ingestion script now caches the previous record to support IP addresses that yield more than one PTR record (field turned into a list when +1). Records will now upsert so MassDNS can be streaming into ES 24/7= 2024-03-07 21:57:10 -05:00
Dionysus b15b3d8241
OCD about formatting again 2024-03-06 15:07:52 -05:00
Dionysus fd617e3c9d
Certstream ingestion improved, still need to test stripping nulls and refactor the index mapping so we only store whats needed 2024-03-06 14:38:34 -05:00
Dionysus 90d6260b99
Updated massdns ingestion script with sentinal value checking and using the ip address as the document id 2024-03-06 14:33:21 -05:00
Dionysus cba51ca2dd
Added a note about NSEC3 hash cracking for zone file data 2024-03-06 14:16:05 -05:00
Dionysus 654e4a8667
Zone file ingestion script now uses the same sentinal value as masscans ingestion, set document id as the domain name to allow updating records if they exist 2024-03-06 14:12:27 -05:00
Dionysus 84f124b23d
Masscan ingestion script updated to use ip:port as the document id to allow updating records that already exist. Added a sentinal value to trigger an EOF when using --watch with FIFO's 2024-03-06 13:26:45 -05:00
Dionysus 5a45be5c60
Updated cause I am OCD about spaces and formatting 2024-03-05 22:29:31 -05:00
Dionysus 598552d34f
Introduction paragraph descriving the project overview added, updated roadmap 2024-03-05 22:26:42 -05:00
Dionysus a4b89e6e5a
Asyncronous refactorization pushed as main version 💯 2024-03-05 22:19:11 -05:00
Dionysus ed547a27f4
Added a recursive JSON explorer to remove empty or nulled keys from a dictionary 2024-03-05 22:15:55 -05:00
Dionysus 4cf976aada
Asyncronous refactorization of the codebase is complete, testing & metrics and then it will be production ready 2024-03-05 21:40:34 -05:00
Dionysus b6fb68ba3a
Asyncronous developed mon ERIS is complete, need to refactor ingestion helpers before pushing this as the main version 2024-03-05 16:47:11 -05:00
Dionysus 31287a99d2
Added certificate transparency monitoring via websockets for ingestion 2024-03-05 11:52:06 -05:00
Dionysus be4938ee6e
Added certificate transparency monitoring via websockets for ingestion 2024-03-05 11:48:53 -05:00
Dionysus c05c48f3fe
Started asyncronous implementation of bulk streaming data, altered ERIS defaults, etc 2024-03-04 17:44:09 -05:00
Dionysus 2ff7ddc869
Added an asyncronous sniffer patch in helpers 2024-02-02 13:21:59 -05:00