Added statistics on WHOIS and RDAP coverage, added a script to compile RDAP/WHOIS servers for all TLDs into a JSON file

This commit is contained in:
Dionysus 2024-03-20 01:23:18 -04:00
parent f6a8744cb3
commit 7da5a00b8f
Signed by: acidvegas
GPG Key ID: EF4B922DB85DC9DE
3 changed files with 5899 additions and 0 deletions

View File

@ -19,6 +19,22 @@ From my experience, it seems every RDAP server has their employed rate limits. W
| `-o`, `--output` | Output file to write successful RDAP data to. | `output.json` | | `-o`, `--output` | Output file to write successful RDAP data to. | `output.json` |
| `-f`, `--failed` | Output file to write failed domains to. (optional) | `failed.txt` | | `-f`, `--failed` | Output file to write failed domains to. (optional) | `failed.txt` |
## WHOIS & RDAP Statistics
By comparing the [root TLDS](https://data.iana.org/TLD/tlds-alpha-by-domain.txt) and the [RDAP Bootstrap](https://data.iana.org/rdap/dns.json) data from IANA, along with querying IANA's [WHOIS](https://www.iana.org/whois) server to find other TLD's WHOIS servers, we can formulate coverage statistics on WHOIS/RDAP availability across all available TLDS:
###### RDAP
| Status | Results |
| ------- | ------------- |
| RDAP | 1,174 *(81%)* |
| No RDAP | 275 *(19%)* |
###### WHOIS
| Status | Results |
| -------- | ------------- |
| WHOIS | 1,253 *(86%)* |
| No WHOIS | 196 *(14%)* |
## Roadmap ## Roadmap
- Explore other RDAP servers from RIRs, registrars, IANA *(see [here](https://data.iana.org/rdap/))* - Explore other RDAP servers from RIRs, registrars, IANA *(see [here](https://data.iana.org/rdap/))*
- Mass domain availability lookups - Mass domain availability lookups

85
whodap.py Normal file
View File

@ -0,0 +1,85 @@
#!/usr/bin/env python
# RDAPr - developed by acidvegas (https://git.acid.vegas/massrdap)
import json
import urllib.request
import socket
import re
whois_rdap = {}
def whois_query(tld: str):
'''
Queries the IANA WHOIS server for TLD information.
:param tld: The top-level domain to query.
'''
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
sock.connect(('whois.iana.org', 43))
sock.sendall((f'{tld}\r\n').encode())
response = b''
while True:
data = sock.recv(4096)
if not data:
break
response += data
return response.decode(errors='replace')
def get_rdap():
'''Fetches RDAP servers from IANA's RDAP Bootstrap file.'''
with urllib.request.urlopen('https://data.iana.org/rdap/dns.json') as response:
data = json.loads(response.read().decode('utf-8'))
for entry in data['services']:
tlds = entry[0]
rdap_url = entry[1][0]
for tld in tlds:
whois_rdap[tld] = {'rdap': rdap_url}
def get_whois():
'''Fetches WHOIS servers from IANA's TLD list.'''
with urllib.request.urlopen('https://data.iana.org/TLD/tlds-alpha-by-domain.txt') as response:
tlds = response.read().decode('utf-8').lower().split('\n')[1:-1]
for tld in tlds:
if tld not in whois_rdap:
whois_rdap[tld] = {'rdap': None}
whois_data = whois_query(tld)
whois_server = None
for line in whois_data.split('\n'):
if 'whois:' in line:
parts = line.split()
if len(parts) > 1:
whois_server = parts[1]
break
if whois_server:
whois_rdap[tld]['whois'] = whois_server
print(f'WHOIS server for {tld}: {whois_server}')
else:
whois_rdap[tld]['whois'] = None
print(f'No WHOIS server for {tld}.')
if __name__ == '__main__':
get_rdap()
TOTAL = len(whois_rdap)
print(f'Found RDAP for {TOTAL:,} TLDs!')
get_whois()
print(f'RDAP is not available for {len(whois_rdap) - TOTAL:,} TLDs.')
whois_rdap = {key: whois_rdap[key] for key in sorted(whois_rdap)}
with open('rdaps.json', 'w') as file:
json.dump(whois_rdap, file, indent=4)

5798
whois_rdap.json Normal file

File diff suppressed because it is too large Load Diff