TOP  

The Truth About Proxy Lists.

Proxies are servers used for anonymous web browsing, data scraping, or to bypass censorships. They provide a gateway to other networks while keeping full anonymity of the original request. Proxies can be private or open/public. 

Proxy lists are simply an updated collection of open proxies, showing IP, country, port, anonymity, uptime, and availability. Lists can be easily copied/pasted into web scraping or SEO tools or manually selected for a particular purpose.


Table of Contents. 

  1. What is a Proxy List?
  2. Why would you need a proxy list?
  3. Types of Proxy Lists
  4. The Details in a Proxy List.
  5. Final Words.

What is a Proxy List? 

It is an inventory or a compilation of open/public proxy servers including (on a basic level), IP addresses and ports. These IPs are not necessarily related. They don’t have to be in the same subnet, location, etc., and the proxies in lists may also vary in accordance with the network protocol, like HTTP, HTTPS, SOCKS, SSL, etc. 

Proxy lists are usually found via websites, but may also be obtained via TXT, XML, or CSV files. A particular “feature” of proxy lists services nowadays, is the daily (sometimes hourly) updates from automatic proxy verification tools. 

Proxy lists usually provide the following (updated) information:

  • Proxy IP address along with the port. 
  • Hostname or domain name. 
  • Location or Country 
  • ISP (in some cases).
  • Protocol (HTTP, HTTPS, SOCKS, etc)
  • Anonymity (Elite, Anonymous, Transparent) 
  • Availability in Uptime (%) 
  • Speed in average response time (ms).

Although some proxy lists are just HTML lists, where you just copy/paste, others will let you search and filter results based on proxy type.

Some advanced sites might even grant you access to proxy list tools like HTML/XML/TXT exporters and information extraction, search filters, proxy scrapers, and anonymity testers to find data leaks. Other popular proxy tools (especially for open and public) will include forums where users can discuss and ask questions about proxies.


Why Would you Need a Proxy List? 

The most popular use case of proxy lists is when clients are looking for many IP addresses (in most cases data-center IPs) to allow a higher degree of anonymity when web scraping, doing market research, or large-scale SEO proxy campaigns

Lists of proxies can be added to web scraping tools such as ScrapeBox (as shown in the screenshot below), to send a request from each IP of the list. The tool takes an IP and automatically rotates it for every request to ensure anonymity and operation without interruptions. 

With proxies from a proxy list, you can either manually rotate IPs, such as the case with browser’s proxies, or let a tool (like Scrapebox) rotate IPs across the list.

Proxy Lists are NOT… 

… pools of IP addresses reserved for proxy servers. These pools are subnets of untouched IPs from the same country assigned to a proxy server. Providers of these proxies usually own the IP, so there is absolutely zero risks of stumbling upon blacklisted IPs. Proxy providers offer IPv6 rotating proxies with unlimited bandwidth. 

To use a rotating proxy, you just need to add an entry node (the IP of a proxy rotator) to grant access and automatically/randomly rotate IPs for each request across the entire IP pool.

Proxy providers may offer a variety of IP addresses from Data Center, Residential, or Mobile.


2. Types of Proxy Lists.

a. Free Proxy List:

Being free doesn’t always mean public and open. Free proxy lists may also be provided by private companies looking to market their paid services or collect data. For that reason, free proxies may be restricted. 

It is very difficult to find a reliable and secure free proxy list. And when you do find them, they quickly change into paid private proxies. Free proxy lists start with a few proxies and limited bandwidth and are likely to quickly funnel you into an upgrade. 

“Forever-free” proxy lists might be unreliable, risky, and slow. In some cases, free proxy lists are contradictory— as the number one reason users look for a proxy, is anonymity, but free proxy providers would often collect activity logs, use it for research, or might even give it away to third parties to use for surveillance or marketing.

Another disadvantage is its lower speed and unreliability. Free proxies are usually used by many at the same time, including those doing intensive web-scraping and web-crawling with bots. 

b. Open Proxy Lists.

Although open proxies follow very similar licensing models as free proxies— their target is different. Obviously, open proxies are free, but they are also public, and usually come without restrictions.

To put it in contrast, “closed proxies” only allow access to users within a certain group defined by a subnet, geographical area, or via authentication. But the list of open proxies allows anyone on the Internet to use the proxies to forward Internet services. 

The advantage of open proxies is that they are free. The disadvantages of open proxies are that they have a lower level of privacy, risk of Malware, and may be subject to blacklists. 

c. Private Proxy Server List:

A private proxy list provider might build its list by finding thousands or hundred thousands of public proxies on the Internet, compiles them into one list, saves them on a database, and keeps updating it constantly. Users use these lists for a while for scraping and bots, and NOT for anonymous web browsing. 

A great advantage of paid proxies is that they usually provide customer service. In case of failure, the need for support, or general troubleshooting, paid proxy list service providers can be contacted immediately.

Bear in mind that even though you are paying for a proxy list, open servers will inevitably fail or be blacklisted. But the fact that you can contact support and ask for help is what makes paid proxy lists a good value for the money. 

Difference between private proxy list and free proxy list? A private proxy list usually provides: 

  • Massive number of proxies.
  • Unlimited traffic and number of flows. 
  • Faster proxies. 
  • Many countries, top anonymity levels, and support for all protocols.
  • Constant updates. Usually new IPs every hour.
  • Efficient caching.
  • Support. 

d. Fresh Proxy Lists. 

Used, public, free, or open proxies have a continuous wide number of users on the server. Between these users, there will always be those noisy neighbors sending intensive traffic through the proxy for things like web scraping or web crawling with bots. 

When proxy information from proxy lists is not updated frequently, users can easily face roadblocks. The noise that these users generate quickly puts proxies’ IPs on blacklists and bans from search engines. 

A fresh proxy list (also known as virgin proxies) is a new list of IP addresses— never been used before. The IPs from these proxy servers are very desirable. As they are untouched IPs, there is no risk of a blacklist or bad reputation behind them. Fresh proxies can be either public or private.

e. Other Types of Proxy Lists.

Some proxy lists are tailored to specifically bypass government restrictions and ISP’s throttling on certain services such as torrenting websites. 

  1. Web proxy lists are powered by server-side scripts like PHPProxy and Glype. These types of proxies only work via the web browser and are usually used to hide your IP by entering the target URL in an embedded form. These are also known as Common Gateway Interface (CGI) proxies.
  2. Mirror or proxy lists for specific URLs. These are very popular among the torrenting communities, such as the pirate proxy lists. Torrent sites that have shut down, have proxy (or mirror) site lists with the same interface with different domains and countries. Users can access banned sites through these proxies.

3. The Details in a Proxy List. 

In the previous section, we defined the types of proxy lists— they can be open, free, private, and fresh. All proxies found on proxy lists are considered “open”… but the way lists are updated, browsed, and filter, is what makes a difference between types. 

Below is an example of a proxy list along with details. 

In the following section, we’ll define the proxy details found in a list. The usual are: 

  • Proxy IP and Port Number.
  • Country/City. 
  • Anonymity level.
  • Protocol (Type).

a. Proxy IP and Port Number (TCP/IP)

The TCP/IP is the suite of network protocols used to interconnect devices. It is based on two protocols, the IP (Network layer) and the TCP (Transport layer). 

In “proxy list world”, the IP is the public IP address of the proxy, the one you use to mask your original source IP. The TCP (Transmission Control Protocol) uses ports to map incoming connections to a specific process running on the proxy. Well-known TCP ports for proxies can be anything from 25, 80 (HTTP), 443 (SSL), 8080 (HTTP), 1080 (SOCKS), or 9000-1000, etc. 

b. Geography-based Proxy List.

Lists of proxies can also be based on geography— a specific country, region, or even city. These proxies share a common IP subnet that belongs to the same country. These proxy lists are specially used for bypassing geo-restricted content. For example, a proxy list including only US-based IPs can be useful for accessing US-based Netflix or Prime content. Although it is very likely that services such as Netflix, quickly identify and block these IPs (unless it is a fresh proxy or a dedicated IP). 

These geography-based proxy lists are also useful for market research, ads testing, or anything that is bound to a geography. 

c. Anonymous Proxy Lists. 

Proxy types based on anonymity level are classified as Transparent (Level 3), Anonymous/Distorting (Level 2), or Elite (Level 3). Transparent proxies leave a trace, Anonymous proxies hide the IP but advertise themselves as proxies, and Elite proxies leave no trace of being a proxy and the source IP. 

d. Protocol-based Proxy Lists.

The most popular proxies based on protocol type are HTTP, HTTPS, SOCKS4, SOCKS5, and CGI. 

  • HTTP Proxy Lists. The HTTP (HyperText Transfer Protocol) proxy lists are the most common type of lists. The HTTP lists contain details of proxies that simply forward HTTP messages from the client to the destination.
  • HTTPS Proxy Lists. HTTPS proxies protect traffic, by making the web browser use the added encryption layer of SSL/TLS. The HTTPS proxy lists only provide information on these types of proxies. These types of lists are also known as SSL proxy lists. 
  • SOCKS proxy list. SOCKets Secure (SOCKS) is an Internet protocol that routes traffic through a “third-party” server, in most cases a firewall. SOCKS routes traffic from any application and protocol. A SOCKS4 proxy list includes information on only SOCKS4 proxies— which don’t support authentication and UDP. A SOCKS5 proxy list, on the other hand, provides an inventory of SOCKS that support various authentication methods and UDP. SOCKS5 proxy lists are popular for users looking for a higher level of security and to use applications that require a higher degree of speed and anonymity, such as torrenting and streaming. 
  • CGI proxy lists. A list of CGI proxy contains information on CGI proxies, which are forwarding proxies used for anonymous web browsing and to access geo-restricted content from sites. 

Final Words.

Open and free proxy lists are continuously blocked and blacklisted. The proxies found on these lists are sometimes used for “noisy activities” such as bypass censorships, botnets, and web scraping— Activities not suited for shared and open proxies.

Although open, free proxy lists are still around, they are now limited for more quiet tasks like anonymous web browsing. If you do find a free proxy list, you should carefully research the provider and their offerings. Bear in mind that open and free proxy lists may compromise your privacy. 

A better alternative for large-scale web scraping, market research, and SEO campaigns are the large quantities of fresh and dedicated pools of IPv4 and IPv6 rotating proxies.

Speak your mind

Leave a Reply

Your email address will not be published. Required fields are marked *