- 資訊首頁(yè) > 開(kāi)發(fā)技術(shù) > 編程語(yǔ)言 >
- c#如何批量抓取免費代理并且驗證有效性
小編給大家分享一下c#如何批量抓取免費代理并且驗證有效性,相信大部分人都還不怎么了解,因此分享這篇文章給大家參考一下,希望大家閱讀完這篇文章后大有收獲,下面讓我們一起去了解一下吧!
1.從哪些網(wǎng)頁(yè)上可以抓取免費的代理IP?
百度一下“免費代理ip”挺多的。
2.代理IP穩定嗎?有什么作用?
這種免費的代理ip時(shí)效性和有效性都不強,上面這三個(gè)免費的代理網(wǎng)站,時(shí)效性大概在十幾秒到1個(gè)小時(shí)不等,一般需要自己處理驗證后使用,提高命中率??蛇m用于隱藏網(wǎng)頁(yè)IP(有些網(wǎng)站還不準使用代理ip,比如豆瓣,其實(shí)挺尷尬的,內容這么貴嗎),一般常用于空間留言、刷網(wǎng)站流量、網(wǎng)賺任務(wù)、批量注冊賬號等,只要沒(méi)有其他限制,需要頻繁更換ip都可以使用。
3.ping通IP就是有效的嗎?如何驗證代理是否有效
好吧,這有點(diǎn)廢話(huà),進(jìn)行端口測試才是最有效的,能ping通并不代表代理有效,不能平通也不一定代理不可用??梢允褂肏ttpWebRequest,也可以使用Scoket,當然HttpWebRequest比Socket連接代理ip、port要慢。
4.一次提取多少代理合適?
代理ip時(shí)效性不強、并且有效性也不高,所以只能從一些代理ip的網(wǎng)站上批量定時(shí)去獲取,有的代理在一分鐘內使用是有限制的,所以說(shuō)限制比較多。
5.http代理和https代理有什么區別?
需要訪(fǎng)問(wèn)https的網(wǎng)站就需要使用https代理了,比如百度,需要訪(fǎng)問(wèn)http的代理,可以使用http。這個(gè)并不是100%的。
檢測代理ip有效性步驟如下:
1.使用HttpWebRequest、HttpWebResponse請求代理ip的網(wǎng)頁(yè),獲取包含代理的網(wǎng)頁(yè)內容
2.使用HtmlAgilityPack或者正則表達式對抓取的內容進(jìn)行截取,保存到代理集合
3.拿到代理集合,多線(xiàn)程發(fā)起http請求,比如訪(fǎng)問(wèn)百度,是否成功,成功則存到里面。
效果圖如下:
使用HttpWebRequest發(fā)起請求
Request.cs如下,主要就是兩個(gè)方法,一個(gè)方法是驗證代理ip是否有效,設置HttpWebRequest的Proxy屬性,請求百度,看到有些文章大多數會(huì )獲取響應的內容,如果內容符合請求的網(wǎng)址則證明代理喲有效,實(shí)際上根據HttpStatusCode 200就可以判斷是否驗證有效。
【注意】建的是控制臺程序,使用了異步,所以還是建.net core吧,c#語(yǔ)言的版本7.1。C#如何在控制臺程序中使用異步
public class Request { /// <summary> /// 驗證代理ip有效性 /// </summary> /// <param name="proxyIp">代理IP</param> /// <param name="proxyPort">代理IP 端口</param> /// <param name="timeout">詳情超時(shí)</param> /// <param name="url">請求的地址</param> /// <param name="success">成功的回調</param> /// <param name="fail">失敗的回調</param> /// <returns></returns> public static async System.Threading.Tasks.Task getAsync(string proxyIp,int proxyPort, int timeout,string url, Action success, Action<string> fail) { System.GC.Collect(); HttpWebRequest request = null; HttpWebResponse response = null; try { request = (HttpWebRequest)WebRequest.Create(url); //HttpWebRequest request = HttpWebRequest.CreateHttp(url); request.Timeout =timeout; request.KeepAlive = false; request.Proxy = new WebProxy(proxyIp,proxyPort); response = await request.GetResponseAsync() as HttpWebResponse; if (response.StatusCode == HttpStatusCode.OK) { success(); } else { fail(response.StatusCode+":"+response.StatusDescription); } } catch (Exception ex) { fail("請求異常"+ex.Message.ToString()); } finally { if (request != null) { request.Abort(); request = null; } if (response != null) { response.Close(); } } } /// <summary> /// 發(fā)起http請求 /// </summary> /// <param name="url"></param> /// <param name="success">成功的回調</param> /// <param name="fail">失敗的回調</param> public static void get(string url,Action<string> success,Action<string> fail) { StreamReader reader = null; Stream stream = null; WebRequest request = null; HttpWebResponse response = null; try { request = WebRequest.Create(url); request.Timeout = 2000; response = (HttpWebResponse)request.GetResponse(); if (response.StatusCode == HttpStatusCode.OK) { stream = response.GetResponseStream(); reader = new StreamReader(stream); string result = reader.ReadToEnd(); success(result); } else { fail(response.StatusCode+":"+response.StatusDescription); } } catch (Exception ex) { fail(ex.ToString()); } finally { if (reader != null) reader.Close(); if (stream != null) stream.Close(); if(response!=null) response.Close(); if(request!=null) request.Abort(); } } }
抓取免費代理,并檢查是否有效
ProxyIpHelper.cs 中主要有四個(gè)方法,檢查ip是否可用CheckProxyIpAsync、抓取xicidaili.com的代理GetXicidailiProxy、抓取ip3366.net的代理GetIp3366Proxy、抓取66ip.cn的代理GetIp3366Proxy。如果想多抓取幾個(gè)網(wǎng)站可以多寫(xiě)幾個(gè)。
public class ProxyIpHelper { private static string address_xicidaili = "http://www.xicidaili.com/wn/{0}"; private static string address_66ip = "http://www.66ip.cn/nmtq.php?getnum=20&isp=0&anonymoustype=0&start=&ports=&export=&ipaddress=&area=1&proxytype=1&api=66ip"; private static string address_ip3366 = "http://www.ip3366.net/?stype=1&page={0}"; /// <summary> /// 檢查代理IP是否可用 /// </summary> /// <param name="ipAddress">ip</param> /// <param name="success">成功的回調</param> /// <param name="fail">失敗的回調</param> /// <returns></returns> public static async Task CheckProxyIpAsync(string ipAddress, Action success, Action<string> fail) { int index = ipAddress.IndexOf(":"); string proxyIp = ipAddress.Substring(0, index); int proxyPort = int.Parse(ipAddress.Substring(index + 1)); await Request.getAsync(proxyIp, proxyPort, 3000, "https://www.baidu.com/", () => { success(); }, (error) => { fail(error); }); } /// <summary> /// 從xicidaili.com網(wǎng)頁(yè)上去獲取代理IP,可以分頁(yè) /// </summary> /// <param name="page"></param> /// <returns></returns> public static List<string> GetXicidailiProxy(int page) { List<string> list = new List<string>(); for (int p = 1; p <= page; p++) { string url = string.Format(address_xicidaili, p); Request.get(url,(docText)=> { if (!string.IsNullOrWhiteSpace(docText)) { HtmlDocument doc = new HtmlDocument(); doc.LoadHtml(docText); var trNodes = doc.DocumentNode.SelectNodes("//table[@id='ip_list']")[0].SelectNodes("./tr"); if (trNodes != null && trNodes.Count > 0) { for (int i = 1; i < trNodes.Count; i++) { var tds = trNodes[i].SelectNodes("./td"); string ipAddress = tds[1].InnerText + ":" + int.Parse(tds[2].InnerText); ; list.Add(ipAddress); } } } },(error)=> { Console.WriteLine(error); }); } return list; } /// <summary> /// 從ip3366.net網(wǎng)頁(yè)上去獲取代理IP,可以分頁(yè) /// </summary> /// <param name="page"></param> /// <returns></returns> public static List<string> GetIp3366Proxy(int page) { List<string> list = new List<string>(); for (int p = 1; p <= page; p++) { string url = string.Format(address_ip3366, p); Request.get(url, (docText) => { if (!string.IsNullOrWhiteSpace(docText)) { HtmlDocument doc = new HtmlDocument(); doc.LoadHtml(docText); var trNodes1 = doc.DocumentNode.SelectNodes("//table")[0]; var trNodes2 = doc.DocumentNode.SelectNodes("//table")[0].SelectSingleNode("//tbody"); var trNodes = doc.DocumentNode.SelectNodes("//table")[0].SelectSingleNode("//tbody").SelectNodes("./tr"); if (trNodes != null && trNodes.Count > 0) { for (int i = 1; i < trNodes.Count; i++) { var tds = trNodes[i].SelectNodes("./td"); if (tds[3].InnerHtml == "HTTPS") { string ipAddress = tds[0].InnerText + ":" + int.Parse(tds[1].InnerText); ; list.Add(ipAddress); } } } } }, (error) => { Console.WriteLine(error); }); } return list; } /// <summary> /// 從66ip.cn中去獲取,不需要分頁(yè) /// </summary> /// <returns></returns> public static List<string> Get66ipProxy() { List<string> list = new List<string>(); Request.get(address_66ip, (docText)=> { int count = 0; if (string.IsNullOrWhiteSpace(docText) == false) { string regex = "\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\.\\d{1,3}\\:\\d{1,5}"; Match mstr = Regex.Match(docText, regex); while (mstr.Success && count < 20) { string tempIp = mstr.Groups[0].Value; list.Add(tempIp); mstr = mstr.NextMatch(); count++; } } }, (error)=> { Console.WriteLine(error); }); return list; } }
使用Timer定時(shí)抓取,并檢查,成功則保存到redis
c#有三種定時(shí)器,這里定時(shí)器是使用System.Threading命名空間, 這個(gè)Timer會(huì )開(kāi)啟新的線(xiàn)程,抓取三個(gè)網(wǎng)頁(yè)定義了三個(gè)Timer對象。每一次抓取都會(huì )保存上一次抓取的集合,檢查前,會(huì )進(jìn)行對比,取出新的集合也就是沒(méi)有重復的那部分。有效性的ip比較低,這里沒(méi)有做統計,如果代碼再優(yōu)化一下,可以做一下統計,看看程序的主入口吧,最終的實(shí)現如下:
class Program { static bool timer_ip3366_isCompleted = true; static bool timer_xicidaili_isCompleted = true; static bool timer_66ip_isCompleted = true; static Timer timer_ip3366, timer_xicidaili, timer_66ip; private static List<string> lastListip3366,lastList66ip,lastListxicidaili;//保存上一次抓取的代理,與下一次進(jìn)行對比,取新的集合進(jìn)行檢查篩選 static async Task Main(string[] args) { System.Net.ServicePointManager.DefaultConnectionLimit = 2000; Console.WriteLine("hellow proxyIp"); Console.ReadLine(); lastList66ip = new List<string>(); lastListip3366 = new List<string>(); lastListxicidaili = new List<string>(); timer_ip3366 = new Timer(async (state) => { await TimerIp3366Async(); }, "processing timer_ip3366 event", 0,1000*30); timer_xicidaili = new Timer(async (state) => { await TimerXicidailiAsync(); }, "processing timer_xicidaili event", 0, 1000 * 60); timer_66ip = new Timer(async (state) => { await Timer66ipAsync(); }, "processing timer_66ip event", 0, 1000*30); Console.ReadLine(); } private static async Task Timer66ipAsync() { if (timer_66ip_isCompleted) { timer_66ip_isCompleted = false; List<string> checkList = new List<string>(); var listProxyIp = ProxyIpHelper.Get66ipProxy(); if (listProxyIp.Count > 0) { Console.ForegroundColor = ConsoleColor.DarkCyan; Console.WriteLine("66ip.cn 抓取到" + listProxyIp.Count + "條記錄,正在對比........."); listProxyIp.ForEach(f => { if (!lastList66ip.Contains(f)) { checkList.Add(f); } }); lastList66ip = listProxyIp; if (checkList.Count > 0) { Console.ForegroundColor = ConsoleColor.DarkCyan; Console.WriteLine("66ip.cn 需要檢查" + checkList.Count + "條記錄,正在進(jìn)行檢測是否有效.........."); for (int i = 0; i < checkList.Count; i++) { string ipAddress = checkList[i]; await ProxyIpHelper.CheckProxyIpAsync(ipAddress, () => { bool insertSuccess = RedisHelper.InsertSet(ipAddress); Console.ForegroundColor = ConsoleColor.White; Console.WriteLine("66ip.cn"); if (insertSuccess) { Console.WriteLine("success" + ipAddress + "任務(wù)編號:" + i + "當前任務(wù)線(xiàn)程:" + Thread.CurrentThread.ManagedThreadId); } Console.WriteLine("重復插入" + ipAddress + "任務(wù)編號:" + i + "當前任務(wù)線(xiàn)程:" + Thread.CurrentThread.ManagedThreadId); }, (error) => { Console.ForegroundColor = ConsoleColor.Green; Console.WriteLine("66ip.cn"); Console.WriteLine("error:" + ipAddress + error + "任務(wù)編號:" + i + "當前任務(wù)線(xiàn)程:" + Thread.CurrentThread.ManagedThreadId); }); } timer_66ip_isCompleted = true; Console.ForegroundColor = ConsoleColor.DarkCyan; Console.WriteLine("66ip.cn" + checkList.Count + "條記錄,已經(jīng)檢測完成,正在進(jìn)行下一次檢查"); } else { timer_66ip_isCompleted = true; Console.ForegroundColor = ConsoleColor.DarkCyan; Console.WriteLine("66ip.cn沒(méi)有需要檢查的代理ip"); } } else { timer_66ip_isCompleted = true; Console.ForegroundColor = ConsoleColor.DarkCyan; Console.WriteLine("66ip.cn沒(méi)有獲取到代理ip"); } } } private static async Task TimerXicidailiAsync() { if (timer_xicidaili_isCompleted) { //取出需要檢查的ip地址,第一次100條則checklist就是100條記錄, //第二次的100條中只有10是和上一次的不重復,則第二次只需要檢查這10條記錄 timer_xicidaili_isCompleted = false; List<string> checkList = new List<string>(); var listProxyIp = ProxyIpHelper.GetXicidailiProxy(1); if (listProxyIp.Count > 0) { Console.WriteLine("xicidaili.com 抓取到" + listProxyIp.Count + "條記錄,正在對比............"); listProxyIp.ForEach(f => { if (!lastListxicidaili.Contains(f)) { checkList.Add(f); } }); lastListxicidaili = listProxyIp; if (checkList.Count > 0) { Console.ForegroundColor = ConsoleColor.DarkCyan; Console.WriteLine("xicidaili.com 需要檢查" + checkList.Count + "條記錄,正在進(jìn)行檢測是否有效.........."); for (int i = 0; i < checkList.Count; i++) { string ipAddress = checkList[i]; await ProxyIpHelper.CheckProxyIpAsync(ipAddress, () => { bool insertSuccess = RedisHelper.InsertSet(ipAddress); Console.ForegroundColor = ConsoleColor.White; Console.WriteLine("xicidaili.com"); if (insertSuccess) { Console.WriteLine("success" + ipAddress + "任務(wù)編號:" + i + "當前任務(wù)線(xiàn)程:" + Thread.CurrentThread.ManagedThreadId); } else Console.WriteLine("重復插入" + ipAddress + "任務(wù)編號:" + i + "當前任務(wù)線(xiàn)程:" + Thread.CurrentThread.ManagedThreadId); }, (error) => { Console.WriteLine("xicidaili.com"); Console.ForegroundColor = ConsoleColor.Red; Console.WriteLine("error:" + ipAddress + error + "任務(wù)編號:" + i + "當前任務(wù)線(xiàn)程:" + Thread.CurrentThread.ManagedThreadId); }); } timer_xicidaili_isCompleted = true; Console.ForegroundColor = ConsoleColor.DarkCyan; Console.WriteLine("xicidaili.com" + checkList.Count + "條記錄,已經(jīng)檢測完成,正在進(jìn)行下一次檢查"); } else { timer_xicidaili_isCompleted = true; Console.ForegroundColor = ConsoleColor.DarkCyan; Console.WriteLine("xicidaili.com沒(méi)有需要檢查的代理ip"); } } else { timer_xicidaili_isCompleted = true; Console.ForegroundColor = ConsoleColor.DarkCyan; Console.WriteLine("xicidaili.com沒(méi)有獲取到代理ip"); } } } private static async Task TimerIp3366Async() { if (timer_ip3366_isCompleted) { timer_ip3366_isCompleted = false; List<string> checkList = new List<string>(); var listProxyIp = ProxyIpHelper.GetIp3366Proxy(4); if (listProxyIp.Count > 0) { Console.ForegroundColor = ConsoleColor.DarkCyan; Console.WriteLine("ip3366.net 抓取到" + listProxyIp.Count + "條記錄,正在進(jìn)行檢測是否有效.........."); listProxyIp.ForEach(f => { if (!lastListip3366.Contains(f)) { checkList.Add(f); } }); lastListip3366 = listProxyIp; if (checkList.Count != 0) { Console.ForegroundColor = ConsoleColor.DarkCyan; Console.WriteLine("ip3366.net 需要檢查" + checkList.Count + "條記錄,正在進(jìn)行檢測是否有效.........."); for (int i = 0; i < checkList.Count; i++) { string ipAddress = checkList[i]; await ProxyIpHelper.CheckProxyIpAsync(ipAddress, () => { bool insertSuccess = RedisHelper.InsertSet(ipAddress); Console.ForegroundColor = ConsoleColor.White; Console.WriteLine("ip3366.net"); if (insertSuccess) { Console.WriteLine("success" + ipAddress + "任務(wù)編號:" + i + "當前任務(wù)線(xiàn)程:" + Thread.CurrentThread.ManagedThreadId); } else { Console.ForegroundColor = ConsoleColor.Red; Console.WriteLine("重復插入" + ipAddress + "任務(wù)編號:" + i + "當前任務(wù)線(xiàn)程:" + Thread.CurrentThread.ManagedThreadId); } }, (error) => { Console.ForegroundColor = ConsoleColor.Yellow; Console.WriteLine("ip3366.net"); Console.WriteLine("error " + ipAddress + "任務(wù)編號:" + i + "當前任務(wù)線(xiàn)程:" + Thread.CurrentThread.ManagedThreadId); }); } timer_ip3366_isCompleted = true; Console.WriteLine("ip3366.net" + checkList.Count + "條記錄,已經(jīng)檢測完成,正在進(jìn)行下一次檢查"); } else { timer_ip3366_isCompleted = true; Console.ForegroundColor = ConsoleColor.DarkCyan; Console.WriteLine("ip3366.net沒(méi)有需要檢查的代理ip"); } } else { timer_ip3366_isCompleted = true; Console.ForegroundColor = ConsoleColor.DarkCyan; Console.WriteLine("ip3366.net沒(méi)有獲取到代理ip"); } } } }
Redis第三庫使用的stackoverflow的 StackExchange.Redis,代理ip不能重復儲存,所以采用的數據結構是Set。存的值非常簡(jiǎn)單就一個(gè)ip加上port,也可以存入更多相關(guān)信息,感覺(jué)沒(méi)必要。即使有這些其他的信息,也很難發(fā)揮作用。RedisHelper.cs如下
public class RedisHelper { private static readonly object Locker = new object(); private static ConnectionMultiplexer _redis; private const string CONNECTTIONSTRING = "127.0.0.1:6379,DefaultDatabase=3"; public const string REDIS_SET_KET_SUCCESS = "set_success_ip"; private static ConnectionMultiplexer Manager { get { if (_redis == null) { lock (Locker) { if (_redis != null) return _redis; _redis = GetManager(); return _redis; } } return _redis; } } private static ConnectionMultiplexer GetManager(string connectionString = null) { if (string.IsNullOrEmpty(connectionString)) { connectionString = CONNECTTIONSTRING; } return ConnectionMultiplexer.Connect(connectionString); } public static bool InsertSet(string value) { var db = Manager.GetDatabase(); return db.SetAdd(REDIS_SET_KET_SUCCESS,value); } }
免責聲明:本站發(fā)布的內容(圖片、視頻和文字)以原創(chuàng )、來(lái)自本網(wǎng)站內容采集于網(wǎng)絡(luò )互聯(lián)網(wǎng)轉載等其它媒體和分享為主,內容觀(guān)點(diǎn)不代表本網(wǎng)站立場(chǎng),如侵犯了原作者的版權,請告知一經(jīng)查實(shí),將立刻刪除涉嫌侵權內容,聯(lián)系我們QQ:712375056,同時(shí)歡迎投稿傳遞力量。
Copyright ? 2009-2022 56dr.com. All Rights Reserved. 特網(wǎng)科技 特網(wǎng)云 版權所有 特網(wǎng)科技 粵ICP備16109289號
域名注冊服務(wù)機構:阿里云計算有限公司(萬(wàn)網(wǎng)) 域名服務(wù)機構:煙臺帝思普網(wǎng)絡(luò )科技有限公司(DNSPod) CDN服務(wù):阿里云計算有限公司 百度云 中國互聯(lián)網(wǎng)舉報中心 增值電信業(yè)務(wù)經(jīng)營(yíng)許可證B2
建議您使用Chrome、Firefox、Edge、IE10及以上版本和360等主流瀏覽器瀏覽本網(wǎng)站