Python 爬取 北京市政府首都之窗信件列表-[信息展示]
日期：2020.01.25
博客期：133
星期六
【代码说明，如果要使用此页代码，必须在本博客页面评论区给予说明】
//博客总体说明
1、准备工作
2、爬取工作
3、数据处理
4、信息展示（本期博客）
大年初一，先给大家说声“过年好”！嗯~说说今天写的部分，因为今天过大年嘛，也没有写什么特别难的东西！就是把上一回博客的内容进行整理，然后根据ECharts的接口进行数据可视化的展示！
下面是项目的目录结构：
嗯！老师说基本的页面是按照我的那个老模板来做的，这样比较省力气！
Java部分：
com.dblink包（连接数据库的包），参考博客：https://www.cnblogs.com/onepersonwholive/p/12168785.html
com.servlet包
package com.servlet; import java.io.IOException;
import java.sql.SQLException;
import java.util.List; import javax.servlet.ServletException;
import javax.servlet.ServletOutputStream;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse; import org.json.JSONArray;
import org.json.JSONObject; import com.dblink.basic.utils.SqlUtils;
import com.dblink.basic.utils.sqlKind.MySql_s;
import com.dblink.basic.utils.user.UserInfo;
import com.dblink.bean.BeanGroup;
import com.dblink.sql.DBLink; @SuppressWarnings("unused")
public class ServletForEveryYear extends HttpServlet{
/**
*
*/
private static final long serialVersionUID = 1L;
//----------------------------------------------------------------------//
public void doPost(HttpServletRequest request,HttpServletResponse response) throws ServletException, IOException
{
request.setCharacterEncoding("utf-8");
response.setCharacterEncoding("utf-8");
response.setContentType("application/json");
response.setHeader("Cache-Control", "no-cache");
response.setHeader("Pragma", "no-cache"); JSONObject jsonobject = new JSONObject(); JSONArray dt = new JSONArray();
JSONArray num = new JSONArray(); DBLink dbLink = new DBLink(new SqlUtils(new MySql_s("edu"),new UserInfo("root","123456")));
BeanGroup bg = null;
try {
bg = dbLink.getSelect("Select * From yeardata").beans;
int leng = bg.size();
for(int i=0;i<leng;++i)
{
dt.put(bg.get(i).get(0));
num.put(bg.get(i).get(1));
}
} catch (SQLException e) {
// Do Nothing ...
}
dbLink.free();
jsonobject.put("YEAR",dt);
jsonobject.put("NUM",num); ServletOutputStream os = response.getOutputStream();
os.write(jsonobject.toString().getBytes());
os.flush();
os.close();
}
//---------------------------------------------------------------------------------//
}
ServletForEveryYear.java
package com.servlet; import java.io.IOException;
import java.sql.SQLException;
import java.util.List; import javax.servlet.ServletException;
import javax.servlet.ServletOutputStream;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse; import org.json.JSONArray;
import org.json.JSONObject; import com.dblink.basic.utils.SqlUtils;
import com.dblink.basic.utils.sqlKind.MySql_s;
import com.dblink.basic.utils.user.UserInfo;
import com.dblink.bean.BeanGroup;
import com.dblink.sql.DBLink; @SuppressWarnings("unused")
public class ServletForResponser extends HttpServlet{
/**
*
*/
private static final long serialVersionUID = 1L;
//----------------------------------------------------------------------//
public void doPost(HttpServletRequest request,HttpServletResponse response) throws ServletException, IOException
{
request.setCharacterEncoding("utf-8");
response.setCharacterEncoding("utf-8");
response.setContentType("application/json");
response.setHeader("Cache-Control", "no-cache");
response.setHeader("Pragma", "no-cache"); JSONObject jsonobject = new JSONObject(); JSONArray dt = new JSONArray();
JSONArray num = new JSONArray(); DBLink dbLink = new DBLink(new SqlUtils(new MySql_s("edu"),new UserInfo("root","123456")));
BeanGroup bg = null;
try {
bg = dbLink.getSelect("Select * From responserdata").beans;
int leng = bg.size();
for(int i=0;i<5;++i)
{
dt.put(bg.get(i).get(0));
num.put(bg.get(i).get(1));
}
} catch (SQLException e) {
// Do Nothing ...
}
dbLink.free();
jsonobject.put("RESPONSER",dt);
jsonobject.put("NUM",num); ServletOutputStream os = response.getOutputStream();
os.write(jsonobject.toString().getBytes());
os.flush();
os.close();
}
//---------------------------------------------------------------------------------//
}
ServletForResponser.java
package com.servlet; import java.io.IOException;
import java.sql.SQLException;
import java.util.List; import javax.servlet.ServletException;
import javax.servlet.ServletOutputStream;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse; import org.json.JSONArray;
import org.json.JSONObject; import com.dblink.basic.utils.SqlUtils;
import com.dblink.basic.utils.sqlKind.MySql_s;
import com.dblink.basic.utils.user.UserInfo;
import com.dblink.bean.BeanGroup;
import com.dblink.sql.DBLink; @SuppressWarnings("unused")
public class ServletForKindData extends HttpServlet{
/**
*
*/
private static final long serialVersionUID = 1L;
//----------------------------------------------------------------------//
public void doPost(HttpServletRequest request,HttpServletResponse response) throws ServletException, IOException
{
request.setCharacterEncoding("utf-8");
response.setCharacterEncoding("utf-8");
response.setContentType("application/json");
response.setHeader("Cache-Control", "no-cache");
response.setHeader("Pragma", "no-cache"); JSONObject jsonobject = new JSONObject(); JSONArray kind = new JSONArray();
JSONArray num = new JSONArray();
JSONArray jsonA = new JSONArray(); DBLink dbLink = new DBLink(new SqlUtils(new MySql_s("edu"),new UserInfo("root","123456")));
BeanGroup bg = null;
try {
bg = dbLink.getSelect("Select * From kinddata").beans;
int leng = bg.size();
for(int i=0;i<leng;++i)
{
JSONObject json = new JSONObject();
json.put("name",bg.get(i).get(0));
json.put("value",bg.get(i).get(1));
kind.put(bg.get(i).get(0));
num.put(bg.get(i).get(1));
jsonA.put(json);
}
} catch (SQLException e) {
// Do Nothing ...
}
dbLink.free();
jsonobject.put("KIND",kind);
jsonobject.put("NUM",num);
jsonobject.put("DATA",jsonA); ServletOutputStream os = response.getOutputStream();
os.write(jsonobject.toString().getBytes());
os.flush();
os.close();
}
//---------------------------------------------------------------------------------//
}
ServletForKindData.java
JSP + JS 部分:
web.xml部分（配置servlet）
<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://xmlns.jcp.org/xml/ns/javaee" xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/web-app_4_0.xsd" id="WebApp_ID" version="4.0">
<display-name>BeiJingEmailPostAndDeal</display-name>
<servlet>
<description>This is the description of my J2EE component</description>
<display-name>This is the display name of my J2EE component</display-name>
<servlet-name>ServletForEveryYear</servlet-name>
<servlet-class>com.servlet.ServletForEveryYear</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>ServletForEveryYear</servlet-name>
<url-pattern>/com/servlet/ServletForEveryYear</url-pattern>
</servlet-mapping>
<servlet>
<description>This is the description of my J2EE component</description>
<display-name>This is the display name of my J2EE component</display-name>
<servlet-name>ServletForResponser</servlet-name>
<servlet-class>com.servlet.ServletForResponser</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>ServletForResponser</servlet-name>
<url-pattern>/com/servlet/ServletForResponser</url-pattern>
</servlet-mapping>
<servlet>
<description>This is the description of my J2EE component</description>
<display-name>This is the display name of my J2EE component</display-name>
<servlet-name>ServletForKindData</servlet-name>
<servlet-class>com.servlet.ServletForKindData</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>ServletForKindData</servlet-name>
<url-pattern>/com/servlet/ServletForKindData</url-pattern>
</servlet-mapping>
<welcome-file-list>
<welcome-file>index.html</welcome-file>
<welcome-file>index.htm</welcome-file>
<welcome-file>index.jsp</welcome-file>
<welcome-file>default.html</welcome-file>
<welcome-file>default.htm</welcome-file>
<welcome-file>default.jsp</welcome-file>
</welcome-file-list>
</web-app>
web.xml
jspFiles
<%@ page language="java" contentType="text/html; charset=utf-8"
pageEncoding="utf-8"%>
<!DOCTYPE html>
<html><!-- xmlns="http://www.w3.org/1999/xhtml" -->
<head>
<!--<meta charset="utf-8" />-->
<meta name="viewport" content="width=device-width, initial-scale=1.0" charset="utf-8"/>
<title>北京市人民政府信件数据分析</title>
<!-- BOOTSTRAP STYLES-->
<link href="../assets/css/bootstrap.css" rel="stylesheet" />
<!-- FONTAWESOME STYLES-->
<link href="../assets/css/font-awesome.css" rel="stylesheet" />
<!-- CUSTOM STYLES-->
<link href="../assets/css/custom.css" rel="stylesheet" />
<!-- PERSONAL FONTS-->
<link href='../cssFiles/basic.css' rel='stylesheet' type='text/css' />
<!-- GOOGLE FONTS-->
<link href='http://fonts.googleapis.com/css?family=Open+Sans' rel='stylesheet' type='text/css' />
</head>
<script src="../jsFiles/jquery/jquery-3.4.1.min.js" charset="utf-8"></script>
<script src="../jsFiles/echarts/echarts.min.js" charset="utf-8"></script>
<script src="../jsFiles/basic.js" charset="utf-8"></script>
<script src="../jsFiles/part1.js" charset="utf-8"></script>
<script src="../jsFiles/part2.js" charset="utf-8"></script>
<script src="../jsFiles/part3.js" charset="utf-8"></script>
<body>
<div id="wrapper">
<div class="navbar navbar-inverse navbar-fixed-top">
<div class="adjust-nav">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".sidebar-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand"><i class="fa fa-square-o "></i> 欢迎您使用本系统</a>
</div>
</div>
</div>
<!-- /. NAV TOP -->
<div class="navbar-default navbar-side"> <!-- nav role="navigation" -->
<div class="sidebar-collapse">
<ul class="nav" id="main-menu">
<li class="text-center user-image-back">
<img src="../assets/img/find_user.png" class="img-responsive" />
</li>
<li>
<a href="#" onclick="makePageToMain()"><i class="fa fa-table "></i>主页</a>
</li>
<li>
<a href="#"><i class="fa fa-edit "></i>可视化界面<span class="fa arrow"></span></a>
<ul class="nav nav-second-level">
<li>
<a href="#" onclick="makePageToPart1()">每年信件数量</a>
</li>
<li>
<a href="#" onclick="makePageToPart2()">每种类型的信件数量</a>
</li>
<li>
<a href="#" onclick="makePageToPart3()">每个政府部门处理的信件数量</a>
</li>
</ul>
</li>
</ul>
</div>
</div>
<!-- /. NAV SIDE -->
<div id="page-wrapper" >
<div id="page-inner">
<div class="row">
<div class="col-md-12">
<h2>主页</h2>
</div>
</div>
<!-- /. ROW -->
<hr />
<!-- /. ROW -->
<br>
<br>
<div id="MessageArea">
<br>
<h3>欢迎您使用本系统，本系统将为您提供三项需求(2020-01-24)</h3>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<span> 如有疑问，也欢迎您访问</span><a href='http://www.beijing.gov.cn/hudong/hdjl/com.web.search.mailList.flow'>原网站</a>以访问更多！
<br>
</div>
</div>
<!-- /. PAGE INNER -->
</div>
<!-- /. PAGE WRAPPER -->
</div>
<!-- /. WRAPPER -->
<!-- SCRIPTS -AT THE BOTOM TO REDUCE THE LOAD TIME-->
<!-- JQUERY SCRIPTS -->
<script src="../assets/js/jquery-1.10.2.js"></script>
<!-- BOOTSTRAP SCRIPTS -->
<script src="../assets/js/bootstrap.min.js"></script>
<!-- METISMENU SCRIPTS -->
<script src="../assets/js/jquery.metisMenu.js"></script>
<!-- CUSTOM SCRIPTS -->
<script src="../assets/js/custom.js"></script>
</body>
</html>
index.jsp
jsFiles
function makePageToPart1()
{
var Area = '';
Area += '<div class="row">';
Area += '<div class="col-md-12">';
Area += '<h2>每年信件数量</h2>';
Area += '</div>';
Area += '</div>';
Area += '<hr />';
Area += '<br>';
Area += '<div id="MessageArea">';
Area += ' <div id="displayArea" style="width:1000px;height:500px;"></div>';
Area += '</div>';
document.getElementById("page-inner").innerHTML = Area;
loadPart1();
}
function loadPart1()
{
var xmlHttp = null;
try{
xmlHttp = new XMLHttpRequest();
} catch (e1) {
try {
xmlHttp = new ActiveXObject("Microsoft.XMLHTTP");
} catch (e2) {
alert("Your browser does not support XMLHTTP!");
return;
}
}
xmlHttp.onreadystatechange = function() {
if (xmlHttp.readyState == 4) {
if (xmlHttp.status == 200)
{
s = xmlHttp.responseText;
var InformationSet = eval('('+s+')');
var leng = InformationSet;
var myChart = echarts.init(document.getElementById('displayArea'));
// 指定图表的配置项和数据
var option = {
title: {
subtext: '单位:条'
},
xAxis: {
name: '年份',
type: 'category',
boundaryGap: false,
axisLine: {onZero: false},
data: InformationSet.YEAR
},
yAxis: {
name: '数量',
type: 'value'
},
tooltip: {
trigger: 'axis'
},
series: [{
name: '数量',
data: InformationSet.NUM,
type: 'line',
markPoint: {
data: [
{type: 'max', name: '最大值'},
{type: 'min', name: '最小值'}
]
}
}]
};
// 使用刚指定的配置项和数据显示图表。
myChart.setOption(option);
}
}
};
var url ="../com/servlet/ServletForEveryYear"; xmlHttp.open("POST", url, true);
xmlHttp.setRequestHeader("Content-Type","application/x-www-form-urlencoded");
xmlHttp.send(null);
}
part1.js
function makePageToPart2()
{
var Area = '';
Area += '<div class="row">';
Area += '<div class="col-md-12">';
Area += '<h2>每种类型的信件数量</h2>';
Area += '</div>';
Area += '</div>';
Area += '<hr />';
Area += '<br>';
Area += '<div id="MessageArea">';
Area += ' <div id="displayArea" style="width:1000px;height:500px;"></div>';
Area += '</div>';
document.getElementById("page-inner").innerHTML = Area;
loadPart2();
}
function loadPart2()
{
var xmlHttp = null;
try{
xmlHttp = new XMLHttpRequest();
} catch (e1) {
try {
xmlHttp = new ActiveXObject("Microsoft.XMLHTTP");
} catch (e2) {
alert("Your browser does not support XMLHTTP!");
return;
}
}
xmlHttp.onreadystatechange = function() {
if (xmlHttp.readyState == 4) {
if (xmlHttp.status == 200)
{
s = xmlHttp.responseText;
var InformationSet = eval('('+s+')');
var myChart = echarts.init(document.getElementById('displayArea'));
var option = {
title: {
text: '每种类型的信件数量',
subtext: '数量',
left: 'center'
},
tooltip: {
trigger: 'item',
formatter: '数量 <br/>{b} : {c} ({d}%)'
},
legend: {
orient: 'vertical',
left: 'left',
data: InformationSet.KIND
},
series: [
{
name: '访问来源',
type: 'pie',
radius: '55%',
center: ['50%', '60%'],
data: InformationSet.DATA,
emphasis: {
itemStyle: {
shadowBlur: 10,
shadowOffsetX: 0,
shadowColor: 'rgba(0, 0, 0, 0.5)'
}
}
}
]
};
// 使用刚指定的配置项和数据显示图表。
myChart.setOption(option);
}
}
};
var url ="../com/servlet/ServletForKindData"; xmlHttp.open("POST", url, true);
xmlHttp.setRequestHeader("Content-Type","application/x-www-form-urlencoded");
xmlHttp.send(null);
}
part2.js
function makePageToPart3()
{
var Area = '';
Area += '<div class="row">';
Area += '<div class="col-md-12">';
Area += '<h2>每个政府部门处理的信件数量</h2>';
Area += '</div>';
Area += '</div>';
Area += '<hr />';
Area += '<br>';
Area += '<div id="MessageArea">';
Area += ' <div id="displayArea" style="width:1200px;height:500px;"></div>';
Area += '</div>';
document.getElementById("page-inner").innerHTML = Area;
loadPart3();
}
function loadPart3()
{
var xmlHttp = null;
try{
xmlHttp = new XMLHttpRequest();
} catch (e1) {
try {
xmlHttp = new ActiveXObject("Microsoft.XMLHTTP");
} catch (e2) {
alert("Your browser does not support XMLHTTP!");
return;
}
}
xmlHttp.onreadystatechange = function() {
if (xmlHttp.readyState == 4) {
if (xmlHttp.status == 200)
{
s = xmlHttp.responseText;
var InformationSet = eval('('+s+')');
var myChart = echarts.init(document.getElementById('displayArea'));
var option = {
xAxis: {
type: 'category',
data: InformationSet.RESPONSER
},
yAxis: {
type: 'value'
},
series: [{
data: InformationSet.NUM,
type: 'bar'
}]
};
// 使用刚指定的配置项和数据显示图表。
myChart.setOption(option);
}
}
};
var url ="../com/servlet/ServletForResponser"; xmlHttp.open("POST", url, true);
xmlHttp.setRequestHeader("Content-Type","application/x-www-form-urlencoded");
xmlHttp.send(null);
}
part3.js
接下来是页面展示：
----[ 主页面 ]
----[ 每年信件数量 ]
----[ 每种类型的信件数量]
----[ 每个政府部门处理的信件数量 ]
Python 爬取 北京市政府首都之窗信件列表-[信息展示]的更多相关文章
- Python 爬取 北京市政府首都之窗信件列表-[Scrapy框架](2020年寒假小目标04)
日期:2020.01.22 博客期:130 星期三 [代码说明,如果要使用此页代码,必须在本博客页面评论区给予说明] //博客总体说明 1.准备工作(本期博客) 2.爬取工作 3.数据处理 4.信息展 ...
- Python 爬取 北京市政府首都之窗信件列表-[后续补充]
日期:2020.01.23 博客期:131 星期四 [本博客的代码如若要使用,请在下方评论区留言,之后再用(就是跟我说一声)] //博客总体说明 1.准备工作 2.爬取工作(本期博客) 3.数据处理 ...
- Python 爬取 北京市政府首都之窗信件列表-[数据处理]
日期:2020.01.24 博客期:132 星期五 [代码说明,如果要使用此页代码,必须在本博客页面评论区给予说明] //博客总体说明 1.准备工作 2.爬取工作 3.数据处理(本期博客) 4.信息展 ...
- Python爬取网址中多个页面的信息
通过上一篇博客了解到爬取数据的操作,但对于存在多个页面的网址来说,使用上一篇博客中的代码爬取下来的资料并不完整.接下来就是讲解该如何爬取之后的页面信息. 一.审查元素 鼠标移至页码处右键,选择检查元素 ...
- python 爬取国家粮食局东北地区玉米收购价格监测信息
#!/usr/bin/python# -*- coding: UTF-8 -*-import reimport sysimport timeimport urllibimport urllib.req ...
- python爬取北京政府信件信息01
python爬取,找到目标地址,开始研究网页代码格式,于是就开始根据之前学的知识进行爬取,出师不利啊,一开始爬取就出现了个问题,这是之前是没有遇到过的,明明地址没问题,就是显示网页不存在,于是就在百度 ...
- Python爬取招聘信息，并且存储到MySQL数据库中
前面一篇文章主要讲述,如何通过Python爬取招聘信息,且爬取的日期为前一天的,同时将爬取的内容保存到数据库中:这篇文章主要讲述如何将python文件压缩成exe可执行文件,供后面的操作. 这系列文章 ...
- Python 爬取所有51VOA网站的Learn a words文本及mp3音频
Python 爬取所有51VOA网站的Learn a words文本及mp3音频 #!/usr/bin/env python # -*- coding: utf-8 -*- #Python 爬取所有5 ...
- python爬取网站数据
开学前接了一个任务,内容是从网上爬取特定属性的数据.正好之前学了python,练练手. 编码问题 因为涉及到中文,所以必然地涉及到了编码的问题,这一次借这个机会算是彻底搞清楚了. 问题要从文字的编码讲 ...
随机推荐
- Matlab geom3d函数注释
Matlab geom3d函数解析 geom3d函数库 geom3d库的目的是处理和可视化三维几何原语,如点.线.平面.多面体等.它提供了操作三维几何原语的底层功能,使得开发更复杂的几何算法变得更加容 ...
- leetcode929 Unique Email Addresses
Every email consists of a local name and a domain name, separated by the @ sign. For example, in ali ...
- 【PAT甲级】1093 Count PAT's (25 分)
题意: 输入一行由大写字母'P','A','T',组成的字符串,输出一共有多少个三元组"PAT"(相对顺序为PAT即可),答案对1e9+7取模. AAAAAccepted code ...
- 基于Amoeba读写分离
Amoeba 原理:amoeba相当于业务员,处理client的读写请求,并将读写请求分开处理.amoeba和master以及slave都有联系,如果是读的请求,amoeba就从slave读取信息反馈 ...
- 将信息存储在claim中，通过扩展AbpSession取出
一.将信息存储到claim中 claims.AddRange(new[] { //新增身份,添加租户id new Claim("RoleName","管理员111&quo ...
- How2j学习java-3下载 ECLIPSE
1.下载并解压Eclipse 下载并解压到e:/eclipse,目录情况如图所示.注: 这个Eclipse是64位的,应该使用本站提供的JDK(64)位,下载地址:JDK. 如果JDK位数和 Ecli ...
- MDC 输出线程信息帮助定位问题
log4j中的%x ---NDC,%X---MDC 即%x NDC.clear();NDC.push(this.toString());%X{first} %X{last}MDC.put(" ...
- Python：集合类型
概念 无序的,不可随机访问的,不可重复的元素集合 与数学中集合的概念类似,可对其进行相关的运算 集合分为可变集合和非可变集合:set 和 frozenset 定义 可变集合 s = {x1, x2, ...
- ZOJ1004 Anagrams by Stack
题目大意:规定 i 为入栈,o 为出栈,现在给两个字符串st1,st2,现在要将st1转化为st2,转化方法是,st1中字符从头开始入栈,并合理出栈构造出st2.请输出所有可能的出入栈步骤. 深度优先 ...
- 【SSM - druid 】配置与使用
web.xml 配置 <!-- druid的监控页面配置开始 --> <servlet> <servlet-name>StatViewServlet</ser ...