20200726_java爬虫_使用HttpClient模拟浏览器发送请求

浏览器获取数据: 打开浏览器 ==> 输入网址 ==> 回车查询 ==> 返回结果 ==> 浏览器显示结果数据

HttpClient获取数据: 创建HttpClient ==> 创建请求方式 ==> 发送请求 ==> 响应结果 ==> 解析结果数据

0. 操作摘要

0.1 添加依赖

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.2</version>
</dependency>

0.2 代码

//1. 打开浏览器  创建httpclient对象
CloseableHttpClient httpClient = HttpClients.createDefault();
//2. 输入网址
HttpGet httpGet = new HttpGet("http://www.baidu.com");
//3. 发送请求
CloseableHttpResponse httpResponse = httpClient.execute(httpGet);
//4. 响应结果
HttpEntity httpEntity = httpResponse.getEntity();
//5. 解析结果
String result = EntityUtils.toString(httpEntity, "utf-8");
System.out.println(result);

1. 实操

1.1 添加依赖

1.1.1 找到 pom.xml 添加依赖

1.1.2 依赖代码

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.2</version>
</dependency>

1.2 添加 TestHttpClient 类

1.2.1 创建类文件 com.aifu.TestHttpClient

1.2.2 添加代码

public static void main(String[] args) throws IOException {
    //1. 打开浏览器  创建httpclient对象
    CloseableHttpClient httpClient = HttpClients.createDefault();
    //2. 输入网址
    HttpGet httpGet = new HttpGet("http://www.baidu.com");
    //3. 发送请求
    CloseableHttpResponse httpResponse = httpClient.execute(httpGet);
    //4. 响应结果
    HttpEntity httpEntity = httpResponse.getEntity();
    //5. 解析结果
    String result = EntityUtils.toString(httpEntity, "utf-8");
    System.out.println(result);
}

1.3 运行

1.3.1 点击绿标运行 或者快捷键 ctrl + alt +F10