Onion Links GitHub

Overview of Onion Link Repositories on GitHub

Developers organize darknet directory lists on GitHub to provide structured access to .onion domains. These repositories serve various purposes, including curation, automation, and validation of Tor hidden services.

Types of Repositories

1. Curated Lists: Repositories like 'oniontree' act as structured directories. They track and maintain curated lists of Tor hidden services, making it easier for users to find reliable .onion links (1). These lists are often updated manually, requiring community contributions to ensure accuracy.

2. Scraping Tools: Automated scripts are available, such as 'DanWin/onion-link-list', designed to compile and manage active Tor hidden service records. These tools can scrape .onion URLs from various sources, streamlining the collection process (2). Command-line tools like 'OnionSearch' enable users to programmatically scrape links across decentralized search engines (3).

3. Validation Scripts: Health checking scripts, such as OnionRoulette, assess the availability of .onion links. These scripts route requests through a local SOCKS5 Tor proxy and check HTTP status codes to verify service uptime (4). This approach helps maintain accurate records of active services.

Directory Structure and Topic Tags

Developers use GitHub Topics to categorize these repositories, allowing for easier navigation. Common tags include "scraping", "Tor", and "hidden services". When creating or organizing a repository, consider the following:

• Structured Metadata: Include relevant details like service name, description, and last checked date.

• Consistent Updates: Regularly validate and update the repository to reflect the current state of .onion links.

• User Contributions: Encourage community members to submit new links or report dead ones to keep the list current.

By understanding the structure and types of onion link repositories available on GitHub, you can effectively navigate and utilize these resources for managing .onion domains.

Top Curated Lists and Hidden Service Directories

Major repository patterns for static .onion link collections include curated lists and automated scraping tools. Open-source repositories like 'oniontree' serve as structured directories for tracking and maintaining curated lists of Tor hidden services (1). These repositories typically feature markdown files or JSON databases for easy access and organization.

Curated Lists

Curated lists provide a simple way to share .onion links. Users can find these lists in repositories that are regularly updated by maintainers. For example, repositories often include:

• Service Name: The name of the Tor hidden service.

• Description: A brief overview of the service.

• Last Checked Date: The date when the service was last validated.

Maintainers typically update these lists manually, ensuring that the links remain accurate and relevant.

Automated Scraping Tools

Repositories like 'DanWin/onion-link-list' provide automated scripts designed to compile and manage active Tor hidden service records (2). These scripts can scrape .onion URLs from various sources, allowing for real-time updates. Command-line tools, such as OnionSearch, enable users to programmatically access multiple decentralized search engines to gather links (3).

Health Checking and Validation

To ensure the reliability of .onion links, health checking scripts are often integrated into these repositories. For example, scripts like OnionRoulette evaluate the availability of hidden services by routing requests through a local SOCKS5 Tor proxy and checking HTTP status codes (4). This process helps maintain an accurate collection of active links.

Structure for Automated Ingestion

Maintainers typically structure markdown files or JSON databases to facilitate automated ingestion. Common practices include:

• Consistent Formatting: Use a standard format for entries to ensure easy parsing.

• Metadata Indexing: Include relevant metadata to enhance searchability.

• Version Control: Utilize Git for tracking changes and updates in the repository.

By following these practices, you can effectively curate and manage .onion links, ensuring that your collections remain up-to-date and organized.

CLI Tools and Scrapers for Onion Discovery

Command-line tools and scrapers are essential for discovering and managing .onion links. One notable tool is OnionSearch, which allows users to scrape .onion URLs from decentralized search engines like Ahmia and Tor66. This tool provides a programmatic approach to gather links efficiently (3).

Multi-processing Behavior

Many scrapers leverage multi-processing to enhance their performance. This approach allows multiple requests to be sent simultaneously, significantly reducing the time required to compile lists of .onion links. For instance, a scraper utilizing Python's multiprocessing library can execute several scraping tasks in parallel, leading to faster data collection and improved throughput.

When implementing multi-processing, consider the following:

• Resource Management: Ensure adequate system resources to handle concurrent requests without causing performance degradation.

• Error Handling: Implement robust error handling to manage potential failures during simultaneous requests.

Output Formats

Scrapers typically produce output in various formats to facilitate easy integration and further analysis. Common output formats include:

• JSON: Useful for structured data, allowing easy access to specific attributes of each .onion link.

• CSV: Ideal for tabular data, making it easy to import into spreadsheet applications or databases.

• Markdown: Suitable for human-readable documentation, enabling users to share collections of links easily.

Example of a Scraping Script

Here’s a simplified snippet of a Python script utilizing multi-processing to scrape .onion links:

import multiprocessing
import requests

def fetch_onion_link(url):
    try:
        response = requests.get(url)
        if response.status_code == 200:
            return url
    except requests.RequestException:
        return None

if __name__ == "__main__":
    urls = ["http://example.onion"] * 10  # Replace with actual .onion links
    with multiprocessing.Pool(processes=5) as pool:
        active_links = pool.map(fetch_onion_link, urls)

This script demonstrates how to fetch multiple .onion links concurrently. Adjust the number of processes based on your system's capabilities.

Health Checking and Metadata Indexing

Health checking scripts, like OnionRoulette, ensure the availability of .onion links by routing requests through a local SOCKS5 Tor proxy and checking HTTP status codes (4). Incorporating uptime validation into your scraping workflow is crucial for maintaining accurate records.

Additionally, consider indexing metadata associated with each .onion link. This practice enhances the discoverability of services and allows for better organization within your repository. Include details such as:

• Service name

• Description

• Last checked date

By utilizing these tools and practices, you can effectively manage and discover .onion links, ensuring your collections remain relevant and useful.

Automated Validation and Health Checking Scripts

The high churn rate of hidden services on the Tor network necessitates effective validation and health checking scripts. Engineers can write scripts that ping .onion addresses via SOCKS5 proxies to verify uptime. This ensures that the links in your repository remain active and reliable.

Health checking scripts, such as OnionRoulette, work by routing requests through a local SOCKS5 Tor proxy, typically set to port 9050. These scripts send HTTP requests to the .onion domains and check the returned status codes. A successful response, such as a 200 OK, indicates that the service is operational (4).

To create an effective health checking script, consider the following steps:

1. Set Up the SOCKS5 Proxy: Configure your script to use a local SOCKS5 proxy for routing requests. This can be done using libraries like requests in Python.

2. Send Requests: Use a loop to send requests to the list of .onion links. Implement error handling to manage potential failures, such as connection issues or timeouts.

3. Check Status Codes: After sending a request, evaluate the HTTP status code returned. You should log responses that indicate service availability or issues.

4. Implement Circuit Rotation: For more effective checking, implement circuit rotation by sending 'SIGNAL NEWNYM' commands to the Tor control port. This fetches fresh exit nodes before every request, reducing the chance of being blocked by the service (5).

5. Output Results: Store the results in a structured format, such as JSON or CSV, for easy analysis and reporting.

Here is a basic example of a health checking script in Python:

import requests

def check_onion_service(url):
    try:
        response = requests.get(url, proxies={"http": "socks5h://localhost:9050"})
        if response.status_code == 200:
            return f"{url} is active."
        else:
            return f"{url} returned status code {response.status_code}."
    except requests.RequestException as e:
        return f"Error checking {url}: {e}"

onion_links = ["http://example1.onion", "http://example2.onion"]  # Replace with actual .onion links
for link in onion_links:
    print(check_onion_service(link))

This script checks each .onion link and outputs its status. Adjust the list of links as needed.

When writing health checking scripts, avoid common configuration issues, such as the "connection refused" error on the Tor control port, which may arise from missing 'ControlPort' or 'CookieAuthentication' settings in your local torrc configuration (6).

By implementing these practices, you can effectively manage the validation of .onion links, ensuring your collections stay current and useful.

Integration with Vector Search and Databases

Modern repositories for scraping .onion metadata leverage vector databases and search engines to enhance querying and categorization. This integration allows for rapid access to curated lists of Tor hidden services. Open-source repositories on GitHub, such as 'oniontree', serve as structured directories for tracking these links (1).

Vector Database Utilization

Vector databases provide efficient indexing capabilities for metadata associated with .onion domains. By transforming textual data into vector representations, these databases facilitate fast similarity searches. For instance, a repository can index metadata such as service names, descriptions, and last checked dates, enabling users to quickly find relevant services.

Automated Scraping and Indexing

Repositories like 'DanWin/onion-link-list' automate the process of compiling active Tor hidden service records (2). These automated scripts not only scrape .onion URLs but also index the associated metadata into a vector database. This allows for real-time updates and ensures that the repository maintains accurate records.

Search Engine Integration

Command-line tools like OnionSearch can scrape .onion URLs from decentralized search engines such as Ahmia and Tor66 (3). By integrating these tools with vector databases, users can perform rapid queries to discover new hidden services. For example, a search for specific keywords can return a ranked list of relevant .onion domains based on their metadata.

Health Checking and Data Validation

Health checking scripts are essential for maintaining the reliability of scraped .onion links. Scripts like OnionRoulette evaluate the availability of hidden services by routing requests through a local SOCKS5 Tor proxy (4). This process not only verifies uptime but can also trigger updates to the metadata stored in the vector database.

Best Practices for Integration

When integrating vector search and databases in your repository, consider the following practices:

• Structured Metadata: Organize metadata in a consistent format for easy indexing.

• Real-Time Updates: Implement automated scripts to ensure that your database reflects the current state of .onion links.

• Robust Error Handling: Prepare for potential failures during scraping to maintain data integrity.

By following these practices, you can effectively index and categorize .onion metadata, enhancing the usability of your repository and streamlining the discovery of Tor hidden services.

Security and Network Isolation Best Practices

When working with untrusted data from GitHub repositories, it is crucial to implement security measures to protect your systems. Running discovery scripts inside isolated Docker containers can significantly reduce risks associated with scraping .onion links.

Docker Container Isolation

Utilizing Docker containers for running your scraping scripts provides an effective way to isolate the execution environment. This minimizes the risk of exposing your host system to potential vulnerabilities present in untrusted code. By default, Docker containers operate in a separate namespace, preventing direct access to the host filesystem and network.

To set up an isolated environment, follow these steps:

1. Create a Docker Image: Build an image that includes all necessary dependencies for your scraping tools. This ensures that the environment is consistent and replicable.

2. Run Containers with Limited Privileges: Use the --cap-drop option to remove unnecessary capabilities from the container. This limits what the container can do on the host system.

3. Network Configuration: Use a custom Docker network to control how your containers communicate with each other and the outside world. This adds another layer of isolation.

SOCKS5 Proxy Routing

Routing your requests through a SOCKS5 proxy enhances security when scraping .onion domains. This method allows you to anonymize your traffic and ensures that requests are handled through the Tor network.

1. Set Up a Local SOCKS5 Proxy: Configure your scripts to use a local SOCKS5 proxy, typically set to port 9050. This proxy should be connected to the Tor network to facilitate .onion requests.

2. Implement Circuit Rotation: For improved anonymity, programmatically send 'SIGNAL NEWNYM' commands to the Tor control port before each request. This will rotate your exit nodes and reduce the risk of tracking (5).

3. Error Handling: Ensure your scripts can gracefully handle connection errors or timeouts. This is particularly important when working with the unpredictable nature of Tor hidden services.

Example Configuration

Here’s a basic example of how to configure a Docker container for your scraping task:

FROM python:3.9-slim

RUN apt-get update && apt-get install -y tor

COPY . /app
WORKDIR /app

RUN pip install -r requirements.txt

CMD ["python", "scraper.py"]

In your scraping script, configure the SOCKS5 proxy as follows:

import requests

proxies = {
    "http": "socks5h://localhost:9050",
    "https": "socks5h://localhost:9050"
}

response = requests.get("http://example.onion", proxies=proxies)

Best Practices Summary

• Use Docker for Isolation: Run all scraping scripts in Docker containers to protect your host system.

• Employ SOCKS5 Proxies: Route requests through a SOCKS5 proxy to ensure anonymity and security.

• Implement Circuit Rotation: Use Tor's control port to rotate exit nodes before requests.

• Handle Errors Gracefully: Prepare your scripts for potential connection issues to maintain stability.

By following these best practices, you can safely parse untrusted data while minimizing the risk to your systems. Implementing security and network isolation measures is essential for any engineer working with .onion links.

Final Takeaways for Managing Onion Links

• Run all scraping and discovery scripts inside isolated Docker containers to protect your host system from untrusted data risks.

• Configure your local Tor proxy correctly on port 9050 and implement circuit rotation to enhance request anonymity and success rates.

• Deploy automated validation tools like OnionRoulette to filter out inactive addresses and maintain accurate link collections.

• Structure your scraped metadata into formats like JSON or CSV and integrate vector databases for rapid querying.

To explore more about setting up robust directory services, check out the Onions Site guide.