An unfortunate reality of cybersecurity is that malicious actors are often almost as clever as benevolent ones, resulting in a constant cat-and-mouse game. Cybersecurity professionals are often on the less advantageous side of the fence as well as they are forced to be reactive to emerging threats, such as when new malware is developed.
Additionally, cybercrime groups will attempt to block researchers and professionals from understanding malware or any other attacks. They may use the same strategy as many other websites – block IP addresses they deem suspicious.
For these and many other reasons, proxies have become one of the foundations for all cybersecurity processes. While they only play a supporting role, that is, enabling researchers to perform their tasks unimpeded, completing those same tasks without proxies would be either much more costly or entirely impossible.
Numerous use cases rolled into one product
Why you can trust TechRadar We spend hours testing every product or service we review, so you can be sure you’re buying the best. Find out more about how we test.
As proxies are a relatively simple technology that provides access to a multitude of IP addresses from various sources and route traffic whenever necessary, it could, at first glance, seem like the uses of such products could be quite limited. Yet, most attacks involve networking, so anything that can change some aspect of it is already useful.
Phishing attacks, for example, mostly come in the form of copycat websites masquerading as legitimate institutions, from banking to ecommerce stores. These websites will be delivered through email, SMS, or other messaging services. Businesses are most commonly attacked through email phishing, as it’s easier to make these messages look like legitimate communication.
Proxies are used in dedicated internal software, usually a form of scrapers, that can automatically scan attachments and visit links that are sent through emails, verifying legitimacy. As the IP addresses acquired from proxies will be innocuous, any blocking or content falsification will not be triggered.
Another common application, although usually deployed as an independent business model, is brand protection or anti-counterfeiting. A lot of products are sold on local marketplaces or geo-restricted websites, normally inaccessible without living in that particular part of the world. With a wide enough range of IP addresses, any restricted website can be visited.
These companies then usually search for illegitimate sellers of branded products, those that are created without acquiring production rights, in other words, fakes. As counterfeits are much more widespread than most people believe, making up to 3.3% of world trade, such usage of proxies protects both consumers and businesses from illegitimate products.
Head of Risk Management at Oxylabs.
Going one step further
While cybersecurity researchers can sometimes be on the back foot when it comes to malicious actors, the time-to-reaction whenever something pops up has significantly decreased. A major part of such an improvement are web scraping technologies, which, in turn, rely on proxies.
Web scraping can collect threat intelligence from both the clear and dark web, enabling cybersecurity professionals to monitor any new occurrences of malicious activity. One of the most common use cases, which many of us have noticed, are data leak discoveries.
Cybersecurity companies can scan the internet for data dumps from company leaks that include emails, passwords, and, potentially, numerous other information points. They can then create databases (or compare with ones of their own) to inform users who have been affected by a leak.
Additionally, web scraping can be used to find newly created websites and verify that they are not used for phishing purposes. Comparing content on a newly discovered and a legitimate website is relatively simple, so web scraping solutions can help cybersecurity companies and financial or government institutions track any attempts at phishing.
Finally, other methods of monitoring are made available through web scraping. Discussions about the development of malware, vulnerability disclosures, and many other publicly available cybersecurity-related actions can be tracked.
It should be noted, however, that web scraping isn’t without faults. There are various data protection legislations enacted throughout the world, which may influence the applicability of such solutions. For example, GDPR is a common pitfall, as any personal data should not be scraped. Private data, such as something behind a login screen, is also beyond the scope of web scraping.
Combining scraping with artificial intelligence
Another extremely promising area for proxies and web scraping is the development of artificial intelligence. In large part, many of the current artificial intelligence implementations are based on machine learning, an extremely data-hungry technology. Web scraping solves the data problem by providing access to nearly all of the publicly available information on the internet.
Machine learning has already been employed by various companies, such as Microsoft. Their Microsoft Defender has been successful in preventing major attacks by detecting new malware automatically.
Anomaly detection, on which Microsoft's success was likely based, is one of the most successful applications for machine learning as it lends itself easily to the technology. Models track networking activity and alert cybersecurity experts whenever something unusual happens, which can then take action if necessary.
Machine learning, however, isn’t limited to anomaly detection. The aforementioned scraping use case for web scraping can be improved by machine learning. Websites usually have some set of criteria, such as URL depth, domain age, keywords, and numerous others, that can be used to verify legitimacy. These factors can be incorporated into machine learning models to automatically detect whether a website has been created for phishing purposes.
Conclusion
Cybersecurity is becoming hard to imagine without the usage of proxies and web scraping. These solutions enable many use cases that allow researchers to become more proactive in their role while, at the same time, reducing their time-to-reaction whenever something happens. Both proxies and web scraping serve as the foundation for many of the current advancements in cybersecurity, and, it is likely, they will continue being fundamental to further developments in the field.
