Full asyncronous library now, able to be used via cli and as a python module with more configurable options

.gitignore

@@ -0,0 +1,7 @@
+.env
+__pycache__
+logs/*
+*.log
+.log. 
+*.egg-info
+dist/

README.md

@@ -2,6 +2,13 @@
 
 The [ICANN Centralized Zone Data Service](https://czds.icann.org) *(CZDS)* allows *approved* users to request and download DNS zone files in bulk, provided they represent a legitimate company or academic institution and their intended use is legal and ethical. Once ICANN approves the request, this tool streamlines the retrieval of extensive domain name system data, facilitating research and security analysis in the realm of internet infrastructure.
 
+## Features
+* Asynchronous downloads with configurable concurrency
+* Support for both CSV and JSON report formats
+* Optional gzip decompression of zone files
+* Environment variable support for credentials
+* Comprehensive error handling and logging
+
 ## Zone Information
 Zone files are updated once every 24 hours, specifically from 00:00 UTC to 06:00 UTC. Access to these zones is granted in increments, and the total time for approval across all zones may extend to a month or longer. It is typical for more than 90% of requested zones to receive approval. Access to certain zone files may require additional application forms with the TLD organization. Please be aware that access to certain zones is time-bound, expiring at the beginning of the following year, or up to a decade after the initial approval has been confirmed.
 
@@ -15,36 +22,53 @@ pip install czds-api
 ```
 
 ## Usage
-###### Command line
+### Command Line Interface
 ```bash
-czds [--username <username> --password <password>] [--concurrency <int>]
+czds [-h] [-u USERNAME] [-p PASSWORD] [-z] [-c CONCURRENCY] [-d] [-k] [-r] [-s] [-f {csv,json}] [-o OUTPUT]
 ```
 
-You can also set the `CZDS_USER` & `CZDS_PASS` environment variables to automatically authenticate:
+#### Arguments
+| Argument              | Description                                  | Default           |
+|-----------------------|----------------------------------------------|-------------------|
+| `-h`, `--help`        | Show help message and exit                   |                   |
+| `-u`, `--username`    | ICANN Username                               | `$CZDS_USER`      |
+| `-p`, `--password`    | ICANN Password                               | `$CZDS_PASS`      |
+| `-z`, `--zones`       | Download zone files                          |                   |
+| `-c`, `--concurrency` | Number of concurrent downloads               | `3`               |
+| `-d`, `--decompress`  | Decompress zone files after download         |                   |
+| `-k`, `--keep`        | Keep original gzip files after decompression |                   |
+| `-r`, `--report`      | Download the zone stats report               |                   |
+| `-s`, `--scrub`       | Scrub username from the report               |                   |
+| `-f`, `--format`      | Report output format (csv/json)              | `csv`             |
+| `-o`, `--output`      | Output directory                             | Current directory |
 
+### Environment Variables
 ```bash
 export CZDS_USER='your_username'
 export CZDS_PASS='your_password'
 ```
 
-###### As a Python module
+### Python Module
 ```python
 import os
-
 from czds import CZDS
 
-CZDS_client = CZDS(username, password)
+async with CZDS(username, password) as client:
+    # Download zone stats report
+    await client.get_report('report.csv', scrub=True, format='json')
     
-CZDS_client.download_report('report.csv')
-
-zone_links = CZDS_client.fetch_zone_links()
-
-os.makedirs('zones', exist_ok=True)
-
-for zone_link in zone_links:
-    CZDS_client.download_zone(zone_link, 'zones')
+    # Download zone files
+    zone_links = await client.fetch_zone_links()
+    await client.download_zones(zone_links, 'zones', concurrency=3, decompress=True)
 ```
 
+## Zone Information
+Zone files are updated once every 24 hours, specifically from 00:00 UTC to 06:00 UTC. Access to these zones is granted in increments, and the total time for approval across all zones may extend to a month or longer. It is typical for more than 90% of requested zones to receive approval. Access to certain zone files may require additional application forms with the TLD organization. Please be aware that access to certain zones is time-bound, expiring at the beginning of the following year, or up to a decade after the initial approval has been confirmed.
+
+At the time of writing this repository, the CZDS offers access to 1,151 zones in total.
+
+1,079 have been approved, 55 are still pending *(after 3 months)*, 10 have been revoked because the TLDs are longer active, and 6 have been denied. Zones that have expired automatically had the expiration extended for me without doing anything, aside from 13 zones that remained expired. I have included a recent [stats file](./extras/stats.csv) directly from my ICANN account.
+
 ## Respects & extras
 While ICANN does have an official [czds-api-client-python](https://github.com/icann/czds-api-client-python) repository, I rewrote it from scratch to be more streamline & included a [POSIX version](./extras/czds) for portability. There is some [official documentation](https://raw.githubusercontent.com/icann/czds-api-client-java/master/docs/ICANN_CZDS_api.pdf) that was referenced in the creation of the POSIX version. Either way, big props to ICANN for allowing me to use the CZDS for research purposes!
 

czds/__init__.py

@@ -5,7 +5,7 @@
 from .client import CZDS
 
 
-__version__ = '1.0.0'
+__version__ = '1.1.0'
 __author__  = 'acidvegas'
 __email__   = 'acid.vegas@acid.vegas'
 __github__  = 'https://github.com/acidvegas/czds'

czds/__main__.py

@@ -3,7 +3,7 @@
 # czds/__main__.py
 
 import argparse
-import concurrent.futures
+import asyncio
 import getpass
 import logging
 import os
@@ -12,67 +12,70 @@ import time
 from .client import CZDS
 
 
-def main(username: str, password: str, concurrency: int) -> None:
-    '''
-    Main function to download all zone files
+async def main():
+    '''Entry point for the command line interface'''
 
-    :param username: ICANN Username
-    :param password: ICANN Password
-    :param concurrency: Number of concurrent downloads
-    '''
-
-    now = time.strftime('%Y-%m-%d')
-    
-    logging.info('Authenticating with ICANN API...')
-
-    CZDS_client = CZDS(username, password)
-
-    logging.debug('Created CZDS client')
-    
-    output_directory = os.path.join(os.getcwd(), 'zones', now)
-    os.makedirs(output_directory, exist_ok=True)
-    
-    logging.info('Fetching zone stats report...')
-
-    try:
-        CZDS_client.download_report(os.path.join(output_directory, '.report.csv'))
-    except Exception as e:
-        raise Exception(f'Failed to download zone stats report: {e}')
-    
-    logging.info('Fetching zone links...')
-
-    try:
-        zone_links = CZDS_client.fetch_zone_links()
-    except Exception as e:
-        raise Exception(f'Failed to fetch zone links: {e}')
-    
-    logging.info(f'Fetched {len(zone_links):,} zone links')
-
-    logging.info('Downloading zone files...')
-    
-    with concurrent.futures.ThreadPoolExecutor(max_workers=concurrency) as executor:
-        future_to_url = {executor.submit(CZDS_client.download_zone, url, output_directory): url for url in sorted(zone_links)}
-        for future in concurrent.futures.as_completed(future_to_url):
-            url = future_to_url[future]
-            try:
-                filepath = future.result()
-                logging.info(f'Completed downloading {url} to file {filepath}')
-            except Exception as e:
-                logging.error(f'{url} generated an exception: {e}')
-
-
-
-if __name__ == '__main__':
+    # Create argument parser
     parser = argparse.ArgumentParser(description='ICANN API for the Centralized Zones Data Service')
+
+    # Authentication
     parser.add_argument('-u', '--username', default=os.getenv('CZDS_USER'), help='ICANN Username')
     parser.add_argument('-p', '--password', default=os.getenv('CZDS_PASS'), help='ICANN Password')
-    parser.add_argument('-c', '--concurrency', type=int, default=3, help='Number of concurrent downloads')
     
+    # Zone download options
+    parser.add_argument('-z', '--zones', action='store_true', help='Download zone files')
+    parser.add_argument('-c', '--concurrency', type=int, default=3, help='Number of concurrent downloads')
+    parser.add_argument('-d', '--decompress', action='store_true', help='Decompress zone files after download')
+    parser.add_argument('-k', '--keep', action='store_true', help='Keep the original gzip files after decompression')
+
+    # Report options
+    parser.add_argument('-r', '--report', action='store_true', help='Download the zone stats report')
+    parser.add_argument('-s', '--scrub', action='store_true', help='Scrub the username from the report')
+    parser.add_argument('-f', '--format', choices=['csv', 'json'], default='csv', help='Report output format')
+
+    # Output options
+    parser.add_argument('-o', '--output', default=os.getcwd(), help='Output directory')
+
+    # Parse arguments
     args = parser.parse_args()
 
     logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(levelname)s - %(message)s')
 
+    # Get username and password
     username = args.username or input('ICANN Username: ')
     password = args.password or getpass.getpass('ICANN Password: ')
 
-    main(username, password, args.concurrency) 
+    # Create output directory
+    now = time.strftime('%Y-%m-%d')
+    output_directory = os.path.join(args.output, 'zones', now)
+    os.makedirs(output_directory, exist_ok=True)
+
+    logging.info('Authenticating with ICANN API...')
+
+    async with CZDS(username, password) as client:
+        # Download zone stats report if requested
+        if args.report:
+            logging.info('Fetching zone stats report...')
+            try:
+                output = os.path.join(output_directory, '.report.csv')
+                await client.get_report(output, scrub=args.scrub, format=args.format)
+                logging.info(f'Zone stats report saved to {output}')
+                return
+            except Exception as e:
+                raise Exception(f'Failed to download zone stats report: {e}')
+        
+        # Download zone files if requested
+        if args.zones:
+            logging.info('Fetching zone links...')
+            try:
+                zone_links = await client.fetch_zone_links()
+            except Exception as e:
+                raise Exception(f'Failed to fetch zone links: {e}')
+
+            logging.info(f'Downloading {len(zone_links):,} zone files...')
+            await client.download_zones(zone_links, output_directory, args.concurrency, decompress=args.decompress, cleanup=not args.keep)
+
+
+
+if __name__ == '__main__':
+    asyncio.run(main()) 

czds/client.py

@@ -2,9 +2,19 @@
 # ICANN API for the Centralized Zones Data Service - developed by acidvegas (https://git.acid.vegas/czds)
 # czds/client.py
 
-import json
+import asyncio
 import os
-import urllib.request
+import gzip
+
+try:
+    import aiohttp
+except ImportError:
+    raise ImportError('missing aiohttp library (pip install aiohttp)')
+
+try:
+    import aiofiles
+except ImportError:
+    raise ImportError('missing aiofiles library (pip install aiofiles)')
 
 
 class CZDS:
@@ -19,86 +29,166 @@ class CZDS:
         '''
 
         self.username = username
-        self.headers  = {'Authorization': f'Bearer {self.authenticate(username, password)}'}
+        self.password = password
+        self.headers  = None # Store the authorization header for reuse
+        self.session  = None # Store the client session for reuse
 
 
-    def authenticate(self, username: str, password: str) -> str:
-        '''
-        Authenticate with the ICANN API and return the access token
+    async def __aenter__(self):
+        '''Async context manager entry'''
 
-        :param username: ICANN Username
-        :param password: ICANN Password
-        '''
+        self.session = aiohttp.ClientSession()
+        self.headers = {'Authorization': f'Bearer {await self.authenticate()}'}
+
+        return self
+
+
+    async def __aexit__(self, exc_type, exc_val, exc_tb):
+        '''Async context manager exit'''
+
+        if self.session:
+            await self.session.close()
+
+
+    async def authenticate(self) -> str:
+        '''Authenticate with the ICANN API and return the access token'''
 
         try:
-            data    = json.dumps({'username': username, 'password': password}).encode('utf-8')
-            headers = {'Content-Type': 'application/json'}
-            request = urllib.request.Request('https://account-api.icann.org/api/authenticate', data=data, headers=headers)
+            data = {'username': self.username, 'password': self.password}
 
-            with urllib.request.urlopen(request) as response:
-                response = response.read().decode('utf-8')
+            async with self.session.post('https://account-api.icann.org/api/authenticate', json=data) as response:
+                if response.status != 200:
+                    raise Exception(f'Authentication failed: {response.status} {await response.text()}')
+
+                result = await response.json()
+
+                return result['accessToken']
 
-            return json.loads(response)['accessToken']
         except Exception as e:
             raise Exception(f'Failed to authenticate with ICANN API: {e}')
 
 
-    def fetch_zone_links(self) -> list:
+    async def fetch_zone_links(self) -> list:
         '''Fetch the list of zone files available for download'''
 
-        request = urllib.request.Request('https://czds-api.icann.org/czds/downloads/links', headers=self.headers)
-        
-        with urllib.request.urlopen(request) as response:
+        async with self.session.get('https://czds-api.icann.org/czds/downloads/links', headers=self.headers) as response:
             if response.status != 200:
-                raise Exception(f'Failed to fetch zone links: {response.status} {response.reason}')
+                raise Exception(f'Failed to fetch zone links: {response.status} {await response.text()}')
 
-            return json.loads(response.read().decode('utf-8'))
+            return await response.json()
 
 
-    def download_report(self, filepath: str):
+    async def get_report(self, filepath: str = None, scrub: bool = True, format: str = 'csv') -> str | dict:
         '''
         Downloads the zone report stats from the API and scrubs the report for privacy
         
         :param filepath: Filepath to save the scrubbed report
+        :param scrub: Whether to scrub the username from the report
+        :param format: Output format ('csv' or 'json')
+        :return: Report content as CSV string or JSON dict
         '''
 
-        request = urllib.request.Request('https://czds-api.icann.org/czds/requests/report', headers=self.headers)
-        
-        with urllib.request.urlopen(request) as response:
+        async with self.session.get('https://czds-api.icann.org/czds/requests/report', headers=self.headers) as response:
             if response.status != 200:
-                raise Exception(f'Failed to download the zone stats report: {response.status} {response.reason}')
+                raise Exception(f'Failed to download the zone stats report: {response.status} {await response.text()}')
 
-            content = response.read().decode('utf-8')
+            content = await response.text()
 
-        with open(filepath, 'w') as file:
-            file.write(content.replace(self.username, 'nobody@no.name'))
+            if scrub:
+                content = content.replace(self.username, 'nobody@no.name')
+
+            if format.lower() == 'json':
+                rows    = [row.split(',') for row in content.strip().split('\n')]
+                header  = rows[0]
+                content = [dict(zip(header, row)) for row in rows[1:]]
+
+            if filepath:
+                async with aiofiles.open(filepath, 'w') as file:
+                    if format.lower() == 'json':
+                        import json
+                        await file.write(json.dumps(content, indent=4))
+                    else:
+                        await file.write(content)
+
+            return content
         
 
-    def download_zone(self, url: str, output_directory: str) -> str:
+    async def gzip_decompress(self, filepath: str, cleanup: bool = True):
+        '''
+        Decompress a gzip file in place
+        
+        :param filepath: Path to the gzip file
+        :param cleanup: Whether to remove the original gzip file after decompression
+        '''
+
+        output_path = filepath[:-3] # Remove .gz extension
+        
+        async with aiofiles.open(filepath, 'rb') as f_in:
+            content = await f_in.read()
+            with gzip.open(content, 'rb') as gz:
+                async with aiofiles.open(output_path, 'wb') as f_out:
+                    await f_out.write(gz.read())
+        
+        if cleanup:
+            os.remove(filepath)
+
+
+    async def download_zone(self, url: str, output_directory: str, decompress: bool = False, cleanup: bool = True, semaphore: asyncio.Semaphore = None):
         '''
         Download a single zone file
         
         :param url: URL to download
         :param output_directory: Directory to save the zone file
+        :param decompress: Whether to decompress the gzip file after download
+        :param cleanup: Whether to remove the original gzip file after decompression
+        :param semaphore: Optional semaphore for controlling concurrency
         '''
 
-        request = urllib.request.Request(url, headers=self.headers)
-
-        with urllib.request.urlopen(request) as response:
+        async def _download():
+            async with self.session.get(url, headers=self.headers) as response:
                 if response.status != 200:
-                raise Exception(f'Failed to download {url}: {response.status} {response.reason}')
+                    raise Exception(f'Failed to download {url}: {response.status} {await response.text()}')
 
-            if not (content_disposition := response.getheader('Content-Disposition')):
+                if not (content_disposition := response.headers.get('Content-Disposition')):
                     raise ValueError('Missing Content-Disposition header')
 
                 filename = content_disposition.split('filename=')[-1].strip('"')
                 filepath = os.path.join(output_directory, filename)
 
-            with open(filepath, 'wb') as file:
+                async with aiofiles.open(filepath, 'wb') as file:
                     while True:
-                    chunk = response.read(1024)
+                        chunk = await response.content.read(8192)
                         if not chunk:
                             break
-                    file.write(chunk)
+                        await file.write(chunk)
+
+                if decompress:
+                    await self.gzip_decompress(filepath, cleanup)
+                    filepath = filepath[:-3] # Remove .gz extension
 
                 return filepath
+
+        if semaphore:
+            async with semaphore:
+                return await _download()
+        else:
+            return await _download()
+
+
+    async def download_zones(self, zone_links: list, output_directory: str, concurrency: int, decompress: bool = False, cleanup: bool = True):
+        '''
+        Download multiple zone files concurrently
+        
+        :param zone_links: List of zone URLs to download
+        :param output_directory: Directory to save the zone files
+        :param concurrency: Number of concurrent downloads
+        :param decompress: Whether to decompress the gzip files after download
+        :param cleanup: Whether to remove the original gzip files after decompression
+        '''
+
+        os.makedirs(output_directory, exist_ok=True)
+
+        semaphore = asyncio.Semaphore(concurrency)
+        tasks     = [self.download_zone(url, output_directory, decompress, cleanup, semaphore) for url in zone_links]
+
+        await asyncio.gather(*tasks)

czds_api.egg-info/PKG-INFO

@@ -0,0 +1,90 @@
+Metadata-Version: 2.2
+Name: czds-api
+Version: 1.0.1
+Summary: ICANN API for the Centralized Zones Data Service
+Home-page: https://github.com/acidvegas/czds
+Author: acidvegas
+Author-email: acid.vegas@acid.vegas
+Project-URL: Bug Tracker, https://github.com/acidvegas/czds/issues
+Project-URL: Documentation, https://github.com/acidvegas/czds#readme
+Project-URL: Source Code, https://github.com/acidvegas/czds
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: ISC License (ISCL)
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.6
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Topic :: Internet
+Classifier: Topic :: Security
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.6
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: project-url
+Dynamic: requires-python
+Dynamic: summary
+
+# ICANN Centralized Zone Data Service API
+
+The [ICANN Centralized Zone Data Service](https://czds.icann.org) *(CZDS)* allows *approved* users to request and download DNS zone files in bulk, provided they represent a legitimate company or academic institution and their intended use is legal and ethical. Once ICANN approves the request, this tool streamlines the retrieval of extensive domain name system data, facilitating research and security analysis in the realm of internet infrastructure.
+
+## Zone Information
+Zone files are updated once every 24 hours, specifically from 00:00 UTC to 06:00 UTC. Access to these zones is granted in increments, and the total time for approval across all zones may extend to a month or longer. It is typical for more than 90% of requested zones to receive approval. Access to certain zone files may require additional application forms with the TLD organization. Please be aware that access to certain zones is time-bound, expiring at the beginning of the following year, or up to a decade after the initial approval has been confirmed.
+
+At the time of writing this repository, the CZDS offers access to 1,151 zones in total.
+
+1,079 have been approved, 55 are still pending *(after 3 months)*, 10 have been revoked because the TLDs are longer active, and 6 have been denied. Zones that have expired automatically had the expiration extended for me without doing anything, aside from 13 zones that remained expired. I have included a recent [stats file](./extras/stats.csv) directly from my ICANN account.
+
+## Installation
+```bash
+pip install czds-api
+```
+
+## Usage
+###### Command line
+```bash
+czds [--username <username> --password <password>] [--concurrency <int>]
+```
+
+You can also set the `CZDS_USER` & `CZDS_PASS` environment variables to automatically authenticate:
+
+```bash
+export CZDS_USER='your_username'
+export CZDS_PASS='your_password'
+```
+
+###### As a Python module
+```python
+import os
+
+from czds import CZDS
+
+CZDS_client = CZDS(username, password)
+
+CZDS_client.download_report('report.csv')
+
+zone_links = CZDS_client.fetch_zone_links()
+
+os.makedirs('zones', exist_ok=True)
+
+for zone_link in zone_links:
+    CZDS_client.download_zone(zone_link, 'zones')
+```
+
+## Respects & extras
+While ICANN does have an official [czds-api-client-python](https://github.com/icann/czds-api-client-python) repository, I rewrote it from scratch to be more streamline & included a [POSIX version](./extras/czds) for portability. There is some [official documentation](https://raw.githubusercontent.com/icann/czds-api-client-java/master/docs/ICANN_CZDS_api.pdf) that was referenced in the creation of the POSIX version. Either way, big props to ICANN for allowing me to use the CZDS for research purposes!
+
+___
+
+###### Mirrors for this repository: [acid.vegas](https://git.acid.vegas/czds) • [SuperNETs](https://git.supernets.org/acidvegas/czds) • [GitHub](https://github.com/acidvegas/czds) • [GitLab](https://gitlab.com/acidvegas/czds) • [Codeberg](https://codeberg.org/acidvegas/czds)

czds_api.egg-info/SOURCES.txt

@@ -0,0 +1,11 @@
+LICENSE
+README.md
+setup.py
+czds/__init__.py
+czds/__main__.py
+czds/client.py
+czds_api.egg-info/PKG-INFO
+czds_api.egg-info/SOURCES.txt
+czds_api.egg-info/dependency_links.txt
+czds_api.egg-info/entry_points.txt
+czds_api.egg-info/top_level.txt

czds_api.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

czds_api.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+czds = czds.__main__:main

czds_api.egg-info/top_level.txt

@@ -0,0 +1 @@
+czds

requirements.txt

@@ -0,0 +1,2 @@
+aiohttp
+aiofiles

setup.py

@@ -11,7 +11,7 @@ with open('README.md', 'r', encoding='utf-8') as fh:
 
 setup(
 	name='czds-api',
-	version='1.0.0',
+	version='1.1.0',
 	author='acidvegas',
 	author_email='acid.vegas@acid.vegas',
 	description='ICANN API for the Centralized Zones Data Service',
@@ -46,4 +46,8 @@ setup(
 			'czds=czds.__main__:main',
 		],
 	},
+	install_requires=[
+		'aiohttp>=3.8.0',
+		'aiofiles>=23.2.1',
+	],
 )