|
|
bd0c8baae3
|
Refactoring RIR ingestions
|
2024-11-22 01:18:28 -05:00 |
|
|
|
124e4b0cf3
|
ECS formatting added to eris, certstream ingestor now caches to prevent duplication, elastic connections properly closed now
|
2024-03-23 22:47:30 -04:00 |
|
|
|
510f7db07e
|
Fixed _doc to doc in record
|
2024-03-23 17:26:55 -04:00 |
|
|
|
c6204be8e1
|
Added more comments and thoughts
|
2024-03-23 15:31:15 -04:00 |
|
|
|
20ee3fcb0f
|
Added a comment about the methodology of determining the zone the data is from.
|
2024-03-23 14:48:45 -04:00 |
|
|
|
e74741c0ad
|
Updated example record comment under source to reflect on the new record structure
|
2024-03-23 14:46:44 -04:00 |
|
|
|
6983b8ebec
|
AXFR detection in zone file input, to determine if the data is form ICANN or an AXFR against a dns (for source tracking)
|
2024-03-23 14:42:51 -04:00 |
|
|
|
a53541c328
|
Added zone field too better querying on specific tlds/zones
|
2024-03-23 14:31:39 -04:00 |
|
|
|
78d7556f09
|
Records stored as a nested type for better querying, added source field so we can identify where the zone data derived (icann, axfr, breach, etc)
|
2024-03-23 13:46:38 -04:00 |
|
|
|
603d005a47
|
Added IXP ingestor, updated all other ingestors
|
2024-03-19 19:00:12 -04:00 |
|
|
|
24850ea976
|
Typo in pypi title
|
2024-03-15 01:26:55 -04:00 |
|
|
|
fe49255f69
|
Added elastic common schema (ecs) logging to file for ingesting eris logs straight into ES
|
2024-03-15 01:25:09 -04:00 |
|
|
|
1ab7199f7d
|
Certstream ingestor now only logs sub-domains since we already ingested zone files. Ignores www. and wildcard domains.
|
2024-03-13 22:34:20 -04:00 |
|
|
|
7f93a4d8de
|
add ingest_rir_transfers.py
|
2024-03-13 20:54:36 -04:00 |
|
|
|
4dc31a5090
|
RIR Transfers ingestor added
|
2024-03-13 20:51:53 -04:00 |
|
|
|
bd735ea8a7
|
Thats how you fucking do it. Delegations stored in ranges now for querying.
|
2024-03-12 21:20:34 -04:00 |
|
|
|
1864f08e4b
|
Fixed import error saying aiofiles instead of aiohttp
|
2024-03-12 18:23:52 -04:00 |
|
|
|
b1fa34f3aa
|
Added anomaly detection to RIR delegations ingestor
|
2024-03-12 18:19:47 -04:00 |
|
|
|
00711fe856
|
Created an ingestor for RIR delegations
|
2024-03-12 17:04:14 -04:00 |
|
|
|
6bb0d4eeef
|
Lowered reconnect time on certstream timeout
|
2024-03-12 00:44:17 -04:00 |
|
|
|
67c7e639a9
|
Renamed default_index to eris-certstream
|
2024-03-11 23:32:37 -04:00 |
|
|
|
1ad7d8a1d3
|
Fixed logger again
|
2024-03-11 23:30:48 -04:00 |
|
|
|
32d5c773e8
|
Fixed missing logging.handlers import
|
2024-03-11 23:07:35 -04:00 |
|
|
|
c3b9cb9e08
|
Fixed logger level
|
2024-03-11 23:06:25 -04:00 |
|
|
|
de3878ef6b
|
Removed _id from certstream yield and renamed ingest_certs.py to ingest_certstream.py
|
2024-03-11 22:46:48 -04:00 |
|
|
|
87f2cf27ea
|
Code cleanup
|
2024-03-11 22:33:18 -04:00 |
|
|
|
b018da4e4d
|
Full source commenting, uniformity in testing function, records stored as a list by default incase an IP address yields multiple PTR records
|
2024-03-11 19:18:03 -04:00 |
|
|
|
6c4ae3e988
|
Many bugs fixed in sniffer and async model.
|
2024-03-08 12:13:57 -05:00 |
|
|
|
d34aa105f1
|
Overall code cleanup
|
2024-03-08 00:07:26 -05:00 |
|
|
|
681e3bd788
|
Fixed issue with ingest_certs and the ingestion function signature. Simple placeholder argument (un-used) added to maintain function uniformity
|
2024-03-07 23:33:20 -05:00 |
|
|
|
45f878285c
|
Testing function added to every ingestor to debug directly. No more --dry-run needed.
|
2024-03-07 23:31:30 -05:00 |
|
|
|
b78b99e060
|
Changed yield to return when sentinal value is recieved through the FIFO
|
2024-03-07 22:57:59 -05:00 |
|
|
|
da065c6889
|
Added a testing function for calling the ingest_massdns.py file directly for debugging
|
2024-03-07 22:55:48 -05:00 |
|
|
|
3c98f8f1f1
|
MassDNS ingestion script now caches the previous record to support IP addresses that yield more than one PTR record (field turned into a list when +1). Records will now upsert so MassDNS can be streaming into ES 24/7=
|
2024-03-07 21:57:44 -05:00 |
|
|
|
9c6beb71ce
|
MassDNS ingestion script now caches the previous record to support IP addresses that yield more than one PTR record (field turned into a list when +1). Records will now upsert so MassDNS can be streaming into ES 24/7=
|
2024-03-07 21:57:10 -05:00 |
|
|
|
b15b3d8241
|
OCD about formatting again
|
2024-03-06 15:07:52 -05:00 |
|
|
|
fd617e3c9d
|
Certstream ingestion improved, still need to test stripping nulls and refactor the index mapping so we only store whats needed
|
2024-03-06 14:38:34 -05:00 |
|
|
|
90d6260b99
|
Updated massdns ingestion script with sentinal value checking and using the ip address as the document id
|
2024-03-06 14:33:21 -05:00 |
|
|
|
cba51ca2dd
|
Added a note about NSEC3 hash cracking for zone file data
|
2024-03-06 14:16:05 -05:00 |
|
|
|
654e4a8667
|
Zone file ingestion script now uses the same sentinal value as masscans ingestion, set document id as the domain name to allow updating records if they exist
|
2024-03-06 14:12:27 -05:00 |
|
|
|
84f124b23d
|
Masscan ingestion script updated to use ip:port as the document id to allow updating records that already exist. Added a sentinal value to trigger an EOF when using --watch with FIFO's
|
2024-03-06 13:26:45 -05:00 |
|
|
|
5a45be5c60
|
Updated cause I am OCD about spaces and formatting
|
2024-03-05 22:29:31 -05:00 |
|
|
|
598552d34f
|
Introduction paragraph descriving the project overview added, updated roadmap
|
2024-03-05 22:26:42 -05:00 |
|
|
|
a4b89e6e5a
|
Asyncronous refactorization pushed as main version 💯
|
2024-03-05 22:19:11 -05:00 |
|
|
|
ed547a27f4
|
Added a recursive JSON explorer to remove empty or nulled keys from a dictionary
|
2024-03-05 22:15:55 -05:00 |
|
|
|
4cf976aada
|
Asyncronous refactorization of the codebase is complete, testing & metrics and then it will be production ready
|
2024-03-05 21:40:34 -05:00 |
|
|
|
b6fb68ba3a
|
Asyncronous developed mon ERIS is complete, need to refactor ingestion helpers before pushing this as the main version
|
2024-03-05 16:47:11 -05:00 |
|
|
|
31287a99d2
|
Added certificate transparency monitoring via websockets for ingestion
|
2024-03-05 11:52:06 -05:00 |
|
|
|
be4938ee6e
|
Added certificate transparency monitoring via websockets for ingestion
|
2024-03-05 11:48:53 -05:00 |
|
|
|
c05c48f3fe
|
Started asyncronous implementation of bulk streaming data, altered ERIS defaults, etc
|
2024-03-04 17:44:09 -05:00 |
|
