Fixed posix script and added stats

README.md

@@ -5,9 +5,9 @@ The [ICANN Centralized Zone Data Service](https://czds.icann.org) *(CZDS)* allow
 ## Zone Information
 Zone files are updated once every 24 hours, specifically from 00:00 UTC to 06:00 UTC. Access to these zones is granted in increments, and the total time for approval across all zones may extend to a month or longer. It is typical for more than 90% of requested zones to receive approval. Access to certain zone files may require additional application forms with the TLD organization. Please be aware that access to certain zones is time-bound, expiring at the beginning of the following year, or up to a decade after the initial approval has been confirmed.
 
-At the time of writing this repository, the CZDS offers access to 1,150 zones in total.
+At the time of writing this repository, the CZDS offers access to 1,151 zones in total.
 
-1,079 have been approved, 55 are still pending *(after 3 months)*, 10 have been revoked because the TLDs are longer active, and 6 have been denied. Zones that have expired automatically had the expiration extended for me without doing anything, aside from 13 zones that remained expired.
+1,079 have been approved, 55 are still pending *(after 3 months)*, 10 have been revoked because the TLDs are longer active, and 6 have been denied. Zones that have expired automatically had the expiration extended for me without doing anything, aside from 13 zones that remained expired. I have included a recent [stats file](./stats_2024-01-31.csv) directly from my ICANN account.
 
 ## Usage
 ### Authentication
@@ -29,7 +29,7 @@ python czds.py [--username <username> --password <password>] [--concurrency <int
 ```
 
 ## Respects
-While ICANN does have an official [czds-api-client-python](https://github.com/icann/czds-api-client-python) repository, I rewrote it from scratch to be more streamline & included a POSIX version for portability. Either way, big props to ICANN for allowing me to use the CZDS for research purposes!
+While ICANN does have an official [czds-api-client-python](https://github.com/icann/czds-api-client-python) repository, I rewrote it from scratch to be more streamline & included a POSIX version for portability. There is some [official documentation](https://raw.githubusercontent.com/icann/czds-api-client-java/master/docs/ICANN_CZDS_api.pdf) that was referenced in the creation of the POSIX version. Either way, big props to ICANN for allowing me to use the CZDS for research purposes!
 
 ___
 

czds

@@ -1,8 +1,6 @@
 #!/bin/bash
 # ICANN API for the Centralized Zones Data Service - developed by acidvegas (https://git.acid.vegas/czds)
-
+# Reference: https://czds.icann.org
-# https://czds.icann.org
-# https://czds.icann.org/sites/default/files/czds-api-documentation.pdf
 
 # Function to authenticate and get access token
 authenticate() {
@@ -22,17 +20,13 @@ authenticate() {
 download_zone() {
 	url="$1"
 	token="$2"
-	filename=$(basename "$url")
+	tld=$(basename "$url" .zone)
-	tld=$(echo "$filename" | cut -d '.' -f 1)
-	filepath="zonefiles/$tld.txt.gz"
-	
 
-	# Create output directory if it does not exist
-	mkdir -p zonefiles
 
 	# Make the GET request and save the response to a file
-	curl --progress-bar -o "$filepath" -H "Authorization: Bearer $token" "$url"
+	echo "Downloading $url..."
-	echo "Downloaded zone file to $filepath"
+	curl --progress-bar -o zonefiles/$tld.txt.gz -H "Authorization: Bearer $token" "$url"
+	echo "Downloaded zone file to zonefiles/$tld.txt.gz"
 }
 
 # Main program starts here
@@ -56,12 +50,12 @@ echo "Fetching zone file links..."
 # Fetch zone links with inline URL and download zone files
 zone_links=$(curl -s -H "Authorization: Bearer $token" "https://czds-api.icann.org/czds/downloads/links" | grep -o 'https://[^"]*')
 
+# Create output directory if it does not exist
+mkdir -p zonefiles
+
 # Download zone files
 for url in $zone_links; do
-	if [ ! -f $filepath ]; then
+	download_zone "$url" "$token"
-		echo "Downloading $url..."
-		download_zone "$url" "$token"
-	fi
 done
 
 echo "All zone files downloaded."

czds.py

@@ -1,11 +1,6 @@
 #!/usr/bin/env python
 # ICANN API for the Centralized Zones Data Service - developed by acidvegas (https://git.acid.vegas/czds)
-
+# Reference: https://czds.icann.org
-'''
-References:
-	- https://czds.icann.org
-	- https://czds.icann.org/sites/default/files/czds-api-documentation.pdf
-'''
 
 import argparse
 import concurrent.futures
@@ -13,7 +8,6 @@ import getpass
 import logging
 import os
 
-
 try:
 	import requests
 except ImportError:
@@ -31,8 +25,10 @@ def authenticate(username: str, password: str) -> str:
 	:param username: ICANN Username
 	:param password: ICANN Password
 	'''
+
 	response = requests.post('https://account-api.icann.org/api/authenticate', json={'username': username, 'password': password})
 	response.raise_for_status()
+
 	return response.json()['accessToken']
 
 
@@ -44,11 +40,13 @@ def download_zone(url: str, token: str, output_directory: str):
 	:param token: ICANN access token
 	:param output_directory: Directory to save the zone file
 	'''
+
 	headers = {'Authorization': f'Bearer {token}'}
 	response = requests.get(url, headers=headers)
 	response.raise_for_status()
 	filename = response.headers.get('Content-Disposition').split('filename=')[-1].strip('"')
 	filepath = os.path.join(output_directory, filename)
+
 	with open(filepath, 'wb') as file:
 		for chunk in response.iter_content(chunk_size=1024):
 			file.write(chunk)
@@ -63,19 +61,24 @@ def main(username: str, password: str, concurrency: int):
 	:param password: ICANN Password
 	:param concurrency: Number of concurrent downloads
 	'''
+
 	token = authenticate(username, password)
 	headers = {'Authorization': f'Bearer {token}'}
-
 	response = requests.get('https://czds-api.icann.org/czds/downloads/links', headers=headers)
 	response.raise_for_status()
 	zone_links = response.json()
 	output_directory = 'zonefiles'
+
 	os.makedirs(output_directory, exist_ok=True)
 
 	with concurrent.futures.ThreadPoolExecutor(max_workers=concurrency) as executor:
+
 		future_to_url = {executor.submit(download_zone, url, token, output_directory): url for url in zone_links}
+
 		for future in concurrent.futures.as_completed(future_to_url):
+
 			url = future_to_url[future]
+
 			try:
 				filepath = future.result()
 				logging.info(f'Completed downloading {url} to file {filepath}')
@@ -96,6 +99,7 @@ if __name__ == '__main__':
 
 	if not username:
 		username = input('ICANN Username: ')
+
 	if not password:
 		password = getpass.getpass('ICANN Password: ')