Added stats CSV from ICANN account, included official documentation reference.

This commit is contained in:
Dionysus 2024-01-30 23:56:31 -05:00
parent 2588ee8d5a
commit ca121e248b
Signed by: acidvegas
GPG Key ID: EF4B922DB85DC9DE
4 changed files with 1167 additions and 15 deletions

View File

@ -5,9 +5,9 @@ The [ICANN Centralized Zone Data Service](https://czds.icann.org) *(CZDS)* allow
## Zone Information
Zone files are updated once every 24 hours, specifically from 00:00 UTC to 06:00 UTC. Access to these zones is granted in increments, and the total time for approval across all zones may extend to a month or longer. It is typical for more than 90% of requested zones to receive approval. Access to certain zone files may require additional application forms with the TLD organization. Please be aware that access to certain zones is time-bound, expiring at the beginning of the following year, or up to a decade after the initial approval has been confirmed.
At the time of writing this repository, the CZDS offers access to 1,150 zones in total.
At the time of writing this repository, the CZDS offers access to 1,151 zones in total.
1,079 have been approved, 55 are still pending *(after 3 months)*, 10 have been revoked because the TLDs are longer active, and 6 have been denied. Zones that have expired automatically had the expiration extended for me without doing anything, aside from 13 zones that remained expired.
1,079 have been approved, 55 are still pending *(after 3 months)*, 10 have been revoked because the TLDs are longer active, and 6 have been denied. Zones that have expired automatically had the expiration extended for me without doing anything, aside from 13 zones that remained expired. I have included a recent [stats file](./stats_2024-01-31.csv) directly from my ICANN account.
## Usage
### Authentication
@ -29,7 +29,7 @@ python czds.py [--username <username> --password <password>] [--concurrency <int
```
## Respects
While ICANN does have an official [czds-api-client-python](https://github.com/icann/czds-api-client-python) repository, I rewrote it from scratch to be more streamline & included a POSIX version for portability. Either way, big props to ICANN for allowing me to use the CZDS for research purposes!
While ICANN does have an official [czds-api-client-python](https://github.com/icann/czds-api-client-python) repository, I rewrote it from scratch to be more streamline & included a POSIX version for portability. There is some [official documentation](https://raw.githubusercontent.com/icann/czds-api-client-java/master/docs/ICANN_CZDS_api.pdf) that was referenced in the creation of the POSIX version. Either way, big props to ICANN for allowing me to use the CZDS for research purposes!
___

5
czds
View File

@ -1,8 +1,6 @@
#!/bin/bash
# ICANN API for the Centralized Zones Data Service - developed by acidvegas (https://git.acid.vegas/czds)
# https://czds.icann.org
# https://czds.icann.org/sites/default/files/czds-api-documentation.pdf
# Reference: https://czds.icann.org
# Function to authenticate and get access token
authenticate() {
@ -25,7 +23,6 @@ download_zone() {
filename=$(basename "$url")
tld=$(echo "$filename" | cut -d '.' -f 1)
filepath="zonefiles/$tld.txt.gz"
# Create output directory if it does not exist
mkdir -p zonefiles

20
czds.py
View File

@ -1,11 +1,6 @@
#!/usr/bin/env python
# ICANN API for the Centralized Zones Data Service - developed by acidvegas (https://git.acid.vegas/czds)
'''
References:
- https://czds.icann.org
- https://czds.icann.org/sites/default/files/czds-api-documentation.pdf
'''
# Reference: https://czds.icann.org
import argparse
import concurrent.futures
@ -13,7 +8,6 @@ import getpass
import logging
import os
try:
import requests
except ImportError:
@ -31,8 +25,10 @@ def authenticate(username: str, password: str) -> str:
:param username: ICANN Username
:param password: ICANN Password
'''
response = requests.post('https://account-api.icann.org/api/authenticate', json={'username': username, 'password': password})
response.raise_for_status()
return response.json()['accessToken']
@ -44,11 +40,13 @@ def download_zone(url: str, token: str, output_directory: str):
:param token: ICANN access token
:param output_directory: Directory to save the zone file
'''
headers = {'Authorization': f'Bearer {token}'}
response = requests.get(url, headers=headers)
response.raise_for_status()
filename = response.headers.get('Content-Disposition').split('filename=')[-1].strip('"')
filepath = os.path.join(output_directory, filename)
with open(filepath, 'wb') as file:
for chunk in response.iter_content(chunk_size=1024):
file.write(chunk)
@ -63,19 +61,24 @@ def main(username: str, password: str, concurrency: int):
:param password: ICANN Password
:param concurrency: Number of concurrent downloads
'''
token = authenticate(username, password)
headers = {'Authorization': f'Bearer {token}'}
response = requests.get('https://czds-api.icann.org/czds/downloads/links', headers=headers)
response.raise_for_status()
zone_links = response.json()
output_directory = 'zonefiles'
os.makedirs(output_directory, exist_ok=True)
with concurrent.futures.ThreadPoolExecutor(max_workers=concurrency) as executor:
future_to_url = {executor.submit(download_zone, url, token, output_directory): url for url in zone_links}
for future in concurrent.futures.as_completed(future_to_url):
url = future_to_url[future]
try:
filepath = future.result()
logging.info(f'Completed downloading {url} to file {filepath}')
@ -96,6 +99,7 @@ if __name__ == '__main__':
if not username:
username = input('ICANN Username: ')
if not password:
password = getpass.getpass('ICANN Password: ')

1151
stats_2024-01-31.csv Normal file

File diff suppressed because it is too large Load Diff