Java爬取并下载酷狗音乐
本文方法及代码仅供学习,仅供学习。
案例:
下载酷狗TOP500歌曲,代码用到的代码库包含:Jsoup、HttpClient、fastJson等。
正文:
1、分析是否可以获取到TOP500歌单
打开酷狗首页,查看TOP500,发现存在分页,每页显示22条歌曲,
发现酷狗的链接如下:
https://www.kugou.com/yy/rank/home/1-8888.html?from=homepage
通过更改链接中的1可以进行分页,所以我们可以通过更改链接地址获取其余的歌曲。
2、分析找到正真的mp3下载地址
点一个歌曲进入播放页面,使用谷歌浏览器的控制台的Elements,搜一下mp3,很轻松就定位到了MP3的位置。
但是使用java访问的时候爬取的html里却没有该mp3的文件地址,那么这肯定是在该页面的位置使用了js来加载mp3,那么刷新下网页,看网页加载了哪些东西,加载的东西有点多,着重看一下js、php的请求,主要是看里面有没有mp3的地址。
最终在列表中找到:
https://wwwapi.kugou.com/yy/index.php?r=play/getdata&callback=jQuery191044492686523157987_1559446927765&hash=458E9B9F362277AC37E9EEF1CB80B535&album_id=18712576&dfid=1ZxQbe0MiP8J09j5tR0Np9IA&mid=9393340fecff864a4d6c4e95099b2be1&platid=4&_=1559446927766
这个请求结果中发现了mp3的完整地址:
那这个js是怎么判断是哪首歌的呢,那么只可能是hash这个参数来决定歌曲的,然后到播放页面里找到这个hash的位置,是在下面的js里:
var dataFromSmarty = [{"hash":"667939C6E784265D541DEEE65AE4F2F8","timelength":"237051","audio_name":"\u767d\u5c0f\u767d - \u6700\u7f8e\u5a5a\u793c","author_name":"\u767d\u5c0f\u767d","song_name":"\u6700\u7f8e\u5a5a\u793c","album_id":0}],//当前页面歌曲信息
playType = "search_single";//当前播放
</script>
在去java爬取该网页,查看能否爬到这个hash,果然,爬取的html里有这段js,到现在mp3的地址也找到了,歌单也找到了,那么下一步就用程序实现就可以了。
3、代码实现
SpiderKugou.java
package com.billy.test; import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONObject;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements; import java.io.IOException;
import java.util.regex.Matcher;
import java.util.regex.Pattern; /**
* 爬取并下载酷狗的歌曲
*/
public class SpiderKugou { private static String filePath; //酷狗地址
private static String LINK; //mp3地址
private static String mp3; static {
filePath = "F:/music/";
LINK = "https://www.kugou.com/yy/rank/home/PAGE-8888.html?from=rank";
mp3 = "https://wwwapi.kugou.com/yy/index.php?r=play/getdata&callback=jQuery19103632090130122796_1558800325111&"
+ "hash=HASH&_=TIME";
} public static void main(String[] args) throws Exception { for(int i = 5 ; i < 23 ; i++){ String url = LINK.replace("PAGE", i + "");
downSong(url);
}
} /**
* 下载歌曲
* @param url
* @throws Exception
*/
private static void downSong(String url) throws Exception{ HttpGetConnect connect = new HttpGetConnect();
String content = connect.connect(url, "utf-8");
HtmlManage html = new HtmlManage();
Document doc = html.manage(content);
Element ele = doc.getElementsByClass("pc_temp_songlist").get(0);
Elements elements = ele.getElementsByTag("li");
for(int i = 0 ; i < elements.size() ; i++){ Element item = elements.get(i);
String title = item.attr("title").trim();
String link = item.getElementsByTag("a").first().attr("href");
downLoad(link,title);
Thread.sleep(1000);
}
} /**
* 下载
* @param url
* @param name
* @throws IOException
*/
private static void downLoad(String url,String name) throws IOException{ String hash = "";
HttpGetConnect connect = new HttpGetConnect();
String content = connect.connect(url, "utf-8"); String regEx = "\"hash\":\"[0-9A-Z]+\"";
// 编译正则表达式
Pattern pattern = Pattern.compile(regEx);
Matcher matcher = pattern.matcher(content);
if (matcher.find()) {
hash = matcher.group();
hash = hash.replace("\"hash\":\"", "");
hash = hash.replace("\"", "");
} String item = mp3.replace("HASH", hash);
item = item.replace("TIME", System.currentTimeMillis() + "");
System.out.println("item:" + item); String mp = connect.connect(item, "utf-8");
mp = mp.substring(mp.indexOf("(") + 1, mp.length() - 3); JSONObject json = JSON.parseObject(mp);
if(Integer.parseInt(json.get("status") + "") != 0){ String playUrl = json.getJSONObject("data").getString("play_url");
FileDownload down = new FileDownload();
down.download(playUrl, filePath + name + ".mp3");
} } }
HttpGetConnect.java
package com.billy.test; import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.http.HttpEntity;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.BasicHttpClientConnectionManager; import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader; /**
* httpclient 工具类
*/
public class HttpGetConnect { /**
* 获取html内容
* @param url
* @param charsetName UTF-8、GB2312
* @return
* @throws IOException
*/
public static String connect(String url,String charsetName) throws IOException{
BasicHttpClientConnectionManager connManager = new BasicHttpClientConnectionManager(); CloseableHttpClient httpclient = HttpClients.custom()
.setConnectionManager(connManager)
.build();
String content = ""; try{
HttpGet httpget = new HttpGet(url); RequestConfig requestConfig = RequestConfig.custom()
.setSocketTimeout(5000)
.setConnectTimeout(50000)
.setConnectionRequestTimeout(50000)
.build();
httpget.setConfig(requestConfig);
httpget.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
httpget.setHeader("Accept-Encoding", "gzip,deflate,sdch");
httpget.setHeader("Accept-Language", "zh-CN,zh;q=0.8");
httpget.setHeader("Connection", "keep-alive");
httpget.setHeader("Upgrade-Insecure-Requests", "1");
httpget.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36");
httpget.setHeader("cache-control", "max-age=0"); httpget.setHeader("Referer","https://www.kugou.com/song/"); //设置cookie
httpget.setHeader("Cookie", "kg_mid=9393340fecff864a4d6c4e95099b2be1;"); CloseableHttpResponse response = httpclient.execute(httpget); int status = response.getStatusLine().getStatusCode();
if (status >= 200 && status < 300) { HttpEntity entity = response.getEntity();
InputStream instream = entity.getContent();
BufferedReader br = new BufferedReader(new InputStreamReader(instream,charsetName));
StringBuffer sbf = new StringBuffer();
String line = null;
while ((line = br.readLine()) != null){
sbf.append(line + "\n");
} br.close();
content = sbf.toString();
} else {
content = "";
} }catch(Exception e){
e.printStackTrace();
}finally{
httpclient.close();
}
log.info("content is " + content);
return content;
} private static Log log = LogFactory.getLog(HttpGetConnect.class);
}
HtmlManage.java
package com.billy.test; import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.select.Elements; import java.io.IOException;
import java.util.ArrayList;
import java.util.List; /**
* html manage 工具类
*/
public class HtmlManage { public Document manage(String html) {
Document doc = Jsoup.parse(html);
return doc;
} public Document manageDirect(String url) throws IOException {
Document doc = Jsoup.connect(url).get();
return doc;
} public List<String> manageHtmlTag(Document doc, String tag) {
List<String> list = new ArrayList<String>(); Elements elements = doc.getElementsByTag(tag);
for (int i = 0; i < elements.size(); i++) {
String str = elements.get(i).html();
list.add(str);
}
return list;
} public List<String> manageHtmlClass(Document doc, String clas) {
List<String> list = new ArrayList<String>(); Elements elements = doc.getElementsByClass(clas);
for (int i = 0; i < elements.size(); i++) {
String str = elements.get(i).html();
list.add(str);
}
return list;
} public List<String> manageHtmlKey(Document doc, String key, String value) {
List<String> list = new ArrayList<String>(); Elements elements = doc.getElementsByAttributeValue(key, value);
for (int i = 0; i < elements.size(); i++) {
String str = elements.get(i).html();
list.add(str);
}
return list;
} private static Log log = LogFactory.getLog(HtmlManage.class);
}
FileDownload.java
package com.billy.test; import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileOutputStream; import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients; /**
* 文件下载工具类
*/
public class FileDownload { /**
* 文件下载
*
* @param url 链接地址
* @param path 要保存的路径及文件名
* @return
*/
public static boolean download(String url, String path) { boolean flag = false; CloseableHttpClient httpclient = HttpClients.createDefault();
RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(2000)
.setConnectTimeout(2000).build(); HttpGet get = new HttpGet(url);
get.setConfig(requestConfig); BufferedInputStream in = null;
BufferedOutputStream out = null;
try {
for (int i = 0; i < 3; i++) {
CloseableHttpResponse result = httpclient.execute(get);
System.out.println(result.getStatusLine());
if (result.getStatusLine().getStatusCode() == 200) {
in = new BufferedInputStream(result.getEntity().getContent());
File file = new File(path);
out = new BufferedOutputStream(new FileOutputStream(file));
byte[] buffer = new byte[1024];
int len = -1;
while ((len = in.read(buffer, 0, 1024)) > -1) {
out.write(buffer, 0, len);
}
flag = true;
break;
} else if (result.getStatusLine().getStatusCode() == 500) {
continue;
}
} } catch (Exception e) {
e.printStackTrace();
flag = false;
} finally {
get.releaseConnection();
try {
if (in != null) {
in.close();
}
if (out != null) {
out.close();
}
} catch (Exception e) {
e.printStackTrace();
flag = false;
}
}
return flag;
} private static Log log = LogFactory.getLog(FileDownload.class);
}
Java爬取并下载酷狗音乐的更多相关文章
- java爬取并下载酷狗TOP500歌曲
是这样的,之前买车送的垃圾记录仪不能用了,这两天狠心买了好点的记录仪,带导航.音乐.蓝牙.4G等功能,寻思,既然有这些功能就利用起来,用4G听歌有点奢侈,就准备去酷狗下点歌听,居然都是需要办会员才能下 ...
- Python 应用爬虫下载酷狗音乐
应用爬虫下载酷狗音乐 首先我们需要进入到这个界面 想要爬取这些歌曲链接,然而这个是一个假的网站,虽然单机右键进行检查能看到这些歌曲的链接,可进行爬取时,却爬取不到这些信息. 这个时候我们就应该换一种思 ...
- Python爬虫下载酷狗音乐
目录 1.Python下载酷狗音乐 1.1.前期准备 1.2.分析 1.2.1.第一步 1.2.2.第二步 1.2.3.第三步 1.2.4.第四步 1.3.代码实现 1.4.运行结果 1.Python ...
- 【Python3爬虫】下载酷狗音乐上的歌曲
经过测试,可以下载要付费下载的歌曲(n_n) 准备工作:Python3.5+Pycharm 使用到的库:requests,re,json,time,fakeuseragent 步骤: 打开酷狗音乐的官 ...
- Python代码搜索并下载酷狗音乐
运行环境: Python3.5+Pycharm 实例代码: import requests,re keyword = input("请输入想要听的歌曲:") url = " ...
- Java爬虫系列之实战:爬取酷狗音乐网 TOP500 的歌曲(附源码)
在前面分享的两篇随笔中分别介绍了HttpClient和Jsoup以及简单的代码案例: Java爬虫系列二:使用HttpClient抓取页面HTML Java爬虫系列三:使用Jsoup解析HTML 今天 ...
- htmlunit+fastjson抓取酷狗音乐 qq音乐链接及下载
上次学了jsoup之后,发现一些动态生成的网页内容是无法抓取的,于是又学习了htmlunit,下面是抓取酷狗音乐与qq音乐链接的例子: 酷狗音乐: import java.io.BufferedInp ...
- python使用beautifulsoup4爬取酷狗音乐
声明:本文仅为技术交流,请勿用于它处. 小编经常在网上听一些音乐但是有一些网站好多音乐都是付费下载的正好我会点爬虫技术,空闲时间写了一份,截止4月底没有问题的,会下载到当前目录,只要按照bs4库就好, ...
- python爬取酷狗音乐排行榜
本文为大家分享了python爬取酷狗音乐排行榜的具体代码,供大家参考,具体内容如下
随机推荐
- [学习笔记] Tangent Distance
Tangent Distance 简介 切空间距离可以用在KNN方法中度量距离,其解决的是图像经过有限变换之后还能否被分类正确,例如.对一张数字为5的手写数字图片,将其膨胀后得到图像p1,此时KNN还 ...
- java实现几种常用排序:选择排序
一.选择排序介绍 选择排序,顾名思义就是用逐个选择的方式来进行排序,逐个选择出数组中的最大(或最小)的元素,直到选择至最后一个元素.此时数组完成了排序. 二.选择排序原理分析 三.选择排序代码实现 / ...
- ubuntu 18.04 64bit下如何源码编译安装anbox
1. 准备工作 1.1 安装gcc 7.x版本 sudo apt-get install gcc-7 -y 1.2 安装依赖的库及其工具 sudo apt install build-essentia ...
- LC 841. Keys and Rooms
There are N rooms and you start in room 0. Each room has a distinct number in 0, 1, 2, ..., N-1, an ...
- LC 963. Minimum Area Rectangle II
Given a set of points in the xy-plane, determine the minimum area of any rectangle formed from these ...
- vue-cli3的安装使用
一.安装vue-cli3 1.全局安装vue-cli 使用命令 cnpm install -g @vue/cli . npm install -g @vue/cli.yarn global add ...
- 基于axis的WebService的案例
1.axis axis是Apache旗下产品,它是一个webservice容器,可被用于创建.部署和运行webservice.网址http://axis.apache.org/,网址上的部分截图如下. ...
- ExtJs6.2环境配置
http://extjs.org.cn/node/831 注意环境变量就是把 创建的项目文件放在C盘中
- app测试自动化之混合APP(之前的三篇为原生APP的操作)
混合开发的App中,经常会有内嵌的H5页面:定位方法与原生APP不同,需要上下文切换 #获取所有上下文contexts=dr.contextsprint(contexts)#打印当前上下文(有点问题应 ...
- KCP - A Fast and Reliable ARQ Protocol
KCP - A Fast and Reliable ARQ Protocol README in English 简介 KCP是一个快速可靠协议,能以比 TCP浪费10%-20%的带宽的代价,换取平均 ...