Python 爬取 北京市政府首都之窗信件列表-[信息展示]
日期：2020.01.25
博客期：133
星期六
【代码说明，如果要使用此页代码，必须在本博客页面评论区给予说明】
//博客总体说明
1、准备工作
2、爬取工作
3、数据处理
4、信息展示（本期博客）
大年初一，先给大家说声“过年好”！嗯~说说今天写的部分，因为今天过大年嘛，也没有写什么特别难的东西！就是把上一回博客的内容进行整理，然后根据ECharts的接口进行数据可视化的展示！
下面是项目的目录结构：
嗯！老师说基本的页面是按照我的那个老模板来做的，这样比较省力气！
Java部分：
com.dblink包（连接数据库的包），参考博客：https://www.cnblogs.com/onepersonwholive/p/12168785.html
com.servlet包
package com.servlet; import java.io.IOException;
import java.sql.SQLException;
import java.util.List; import javax.servlet.ServletException;
import javax.servlet.ServletOutputStream;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse; import org.json.JSONArray;
import org.json.JSONObject; import com.dblink.basic.utils.SqlUtils;
import com.dblink.basic.utils.sqlKind.MySql_s;
import com.dblink.basic.utils.user.UserInfo;
import com.dblink.bean.BeanGroup;
import com.dblink.sql.DBLink; @SuppressWarnings("unused")
public class ServletForEveryYear extends HttpServlet{
/**
*
*/
private static final long serialVersionUID = 1L;
//----------------------------------------------------------------------//
public void doPost(HttpServletRequest request,HttpServletResponse response) throws ServletException, IOException
{
request.setCharacterEncoding("utf-8");
response.setCharacterEncoding("utf-8");
response.setContentType("application/json");
response.setHeader("Cache-Control", "no-cache");
response.setHeader("Pragma", "no-cache"); JSONObject jsonobject = new JSONObject(); JSONArray dt = new JSONArray();
JSONArray num = new JSONArray(); DBLink dbLink = new DBLink(new SqlUtils(new MySql_s("edu"),new UserInfo("root","123456")));
BeanGroup bg = null;
try {
bg = dbLink.getSelect("Select * From yeardata").beans;
int leng = bg.size();
for(int i=0;i<leng;++i)
{
dt.put(bg.get(i).get(0));
num.put(bg.get(i).get(1));
}
} catch (SQLException e) {
// Do Nothing ...
}
dbLink.free();
jsonobject.put("YEAR",dt);
jsonobject.put("NUM",num); ServletOutputStream os = response.getOutputStream();
os.write(jsonobject.toString().getBytes());
os.flush();
os.close();
}
//---------------------------------------------------------------------------------//
}
ServletForEveryYear.java
package com.servlet; import java.io.IOException;
import java.sql.SQLException;
import java.util.List; import javax.servlet.ServletException;
import javax.servlet.ServletOutputStream;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse; import org.json.JSONArray;
import org.json.JSONObject; import com.dblink.basic.utils.SqlUtils;
import com.dblink.basic.utils.sqlKind.MySql_s;
import com.dblink.basic.utils.user.UserInfo;
import com.dblink.bean.BeanGroup;
import com.dblink.sql.DBLink; @SuppressWarnings("unused")
public class ServletForResponser extends HttpServlet{
/**
*
*/
private static final long serialVersionUID = 1L;
//----------------------------------------------------------------------//
public void doPost(HttpServletRequest request,HttpServletResponse response) throws ServletException, IOException
{
request.setCharacterEncoding("utf-8");
response.setCharacterEncoding("utf-8");
response.setContentType("application/json");
response.setHeader("Cache-Control", "no-cache");
response.setHeader("Pragma", "no-cache"); JSONObject jsonobject = new JSONObject(); JSONArray dt = new JSONArray();
JSONArray num = new JSONArray(); DBLink dbLink = new DBLink(new SqlUtils(new MySql_s("edu"),new UserInfo("root","123456")));
BeanGroup bg = null;
try {
bg = dbLink.getSelect("Select * From responserdata").beans;
int leng = bg.size();
for(int i=0;i<5;++i)
{
dt.put(bg.get(i).get(0));
num.put(bg.get(i).get(1));
}
} catch (SQLException e) {
// Do Nothing ...
}
dbLink.free();
jsonobject.put("RESPONSER",dt);
jsonobject.put("NUM",num); ServletOutputStream os = response.getOutputStream();
os.write(jsonobject.toString().getBytes());
os.flush();
os.close();
}
//---------------------------------------------------------------------------------//
}
ServletForResponser.java
package com.servlet; import java.io.IOException;
import java.sql.SQLException;
import java.util.List; import javax.servlet.ServletException;
import javax.servlet.ServletOutputStream;
import javax.servlet.annotation.WebServlet;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse; import org.json.JSONArray;
import org.json.JSONObject; import com.dblink.basic.utils.SqlUtils;
import com.dblink.basic.utils.sqlKind.MySql_s;
import com.dblink.basic.utils.user.UserInfo;
import com.dblink.bean.BeanGroup;
import com.dblink.sql.DBLink; @SuppressWarnings("unused")
public class ServletForKindData extends HttpServlet{
/**
*
*/
private static final long serialVersionUID = 1L;
//----------------------------------------------------------------------//
public void doPost(HttpServletRequest request,HttpServletResponse response) throws ServletException, IOException
{
request.setCharacterEncoding("utf-8");
response.setCharacterEncoding("utf-8");
response.setContentType("application/json");
response.setHeader("Cache-Control", "no-cache");
response.setHeader("Pragma", "no-cache"); JSONObject jsonobject = new JSONObject(); JSONArray kind = new JSONArray();
JSONArray num = new JSONArray();
JSONArray jsonA = new JSONArray(); DBLink dbLink = new DBLink(new SqlUtils(new MySql_s("edu"),new UserInfo("root","123456")));
BeanGroup bg = null;
try {
bg = dbLink.getSelect("Select * From kinddata").beans;
int leng = bg.size();
for(int i=0;i<leng;++i)
{
JSONObject json = new JSONObject();
json.put("name",bg.get(i).get(0));
json.put("value",bg.get(i).get(1));
kind.put(bg.get(i).get(0));
num.put(bg.get(i).get(1));
jsonA.put(json);
}
} catch (SQLException e) {
// Do Nothing ...
}
dbLink.free();
jsonobject.put("KIND",kind);
jsonobject.put("NUM",num);
jsonobject.put("DATA",jsonA); ServletOutputStream os = response.getOutputStream();
os.write(jsonobject.toString().getBytes());
os.flush();
os.close();
}
//---------------------------------------------------------------------------------//
}
ServletForKindData.java
JSP + JS 部分:
web.xml部分（配置servlet）
<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://xmlns.jcp.org/xml/ns/javaee" xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee http://xmlns.jcp.org/xml/ns/javaee/web-app_4_0.xsd" id="WebApp_ID" version="4.0">
<display-name>BeiJingEmailPostAndDeal</display-name>
<servlet>
<description>This is the description of my J2EE component</description>
<display-name>This is the display name of my J2EE component</display-name>
<servlet-name>ServletForEveryYear</servlet-name>
<servlet-class>com.servlet.ServletForEveryYear</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>ServletForEveryYear</servlet-name>
<url-pattern>/com/servlet/ServletForEveryYear</url-pattern>
</servlet-mapping>
<servlet>
<description>This is the description of my J2EE component</description>
<display-name>This is the display name of my J2EE component</display-name>
<servlet-name>ServletForResponser</servlet-name>
<servlet-class>com.servlet.ServletForResponser</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>ServletForResponser</servlet-name>
<url-pattern>/com/servlet/ServletForResponser</url-pattern>
</servlet-mapping>
<servlet>
<description>This is the description of my J2EE component</description>
<display-name>This is the display name of my J2EE component</display-name>
<servlet-name>ServletForKindData</servlet-name>
<servlet-class>com.servlet.ServletForKindData</servlet-class>
</servlet>
<servlet-mapping>
<servlet-name>ServletForKindData</servlet-name>
<url-pattern>/com/servlet/ServletForKindData</url-pattern>
</servlet-mapping>
<welcome-file-list>
<welcome-file>index.html</welcome-file>
<welcome-file>index.htm</welcome-file>
<welcome-file>index.jsp</welcome-file>
<welcome-file>default.html</welcome-file>
<welcome-file>default.htm</welcome-file>
<welcome-file>default.jsp</welcome-file>
</welcome-file-list>
</web-app>
web.xml
jspFiles
<%@ page language="java" contentType="text/html; charset=utf-8"
pageEncoding="utf-8"%>
<!DOCTYPE html>
<html><!-- xmlns="http://www.w3.org/1999/xhtml" -->
<head>
<!--<meta charset="utf-8" />-->
<meta name="viewport" content="width=device-width, initial-scale=1.0" charset="utf-8"/>
<title>北京市人民政府信件数据分析</title>
<!-- BOOTSTRAP STYLES-->
<link href="../assets/css/bootstrap.css" rel="stylesheet" />
<!-- FONTAWESOME STYLES-->
<link href="../assets/css/font-awesome.css" rel="stylesheet" />
<!-- CUSTOM STYLES-->
<link href="../assets/css/custom.css" rel="stylesheet" />
<!-- PERSONAL FONTS-->
<link href='../cssFiles/basic.css' rel='stylesheet' type='text/css' />
<!-- GOOGLE FONTS-->
<link href='http://fonts.googleapis.com/css?family=Open+Sans' rel='stylesheet' type='text/css' />
</head>
<script src="../jsFiles/jquery/jquery-3.4.1.min.js" charset="utf-8"></script>
<script src="../jsFiles/echarts/echarts.min.js" charset="utf-8"></script>
<script src="../jsFiles/basic.js" charset="utf-8"></script>
<script src="../jsFiles/part1.js" charset="utf-8"></script>
<script src="../jsFiles/part2.js" charset="utf-8"></script>
<script src="../jsFiles/part3.js" charset="utf-8"></script>
<body>
<div id="wrapper">
<div class="navbar navbar-inverse navbar-fixed-top">
<div class="adjust-nav">
<div class="navbar-header">
<button type="button" class="navbar-toggle" data-toggle="collapse" data-target=".sidebar-collapse">
<span class="icon-bar"></span>
<span class="icon-bar"></span>
<span class="icon-bar"></span>
</button>
<a class="navbar-brand"><i class="fa fa-square-o "></i> 欢迎您使用本系统</a>
</div>
</div>
</div>
<!-- /. NAV TOP -->
<div class="navbar-default navbar-side"> <!-- nav role="navigation" -->
<div class="sidebar-collapse">
<ul class="nav" id="main-menu">
<li class="text-center user-image-back">
<img src="../assets/img/find_user.png" class="img-responsive" />
</li>
<li>
<a href="#" onclick="makePageToMain()"><i class="fa fa-table "></i>主页</a>
</li>
<li>
<a href="#"><i class="fa fa-edit "></i>可视化界面<span class="fa arrow"></span></a>
<ul class="nav nav-second-level">
<li>
<a href="#" onclick="makePageToPart1()">每年信件数量</a>
</li>
<li>
<a href="#" onclick="makePageToPart2()">每种类型的信件数量</a>
</li>
<li>
<a href="#" onclick="makePageToPart3()">每个政府部门处理的信件数量</a>
</li>
</ul>
</li>
</ul>
</div>
</div>
<!-- /. NAV SIDE -->
<div id="page-wrapper" >
<div id="page-inner">
<div class="row">
<div class="col-md-12">
<h2>主页</h2>
</div>
</div>
<!-- /. ROW -->
<hr />
<!-- /. ROW -->
<br>
<br>
<div id="MessageArea">
<br>
<h3>欢迎您使用本系统，本系统将为您提供三项需求(2020-01-24)</h3>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<br>
<span> 如有疑问，也欢迎您访问</span><a href='http://www.beijing.gov.cn/hudong/hdjl/com.web.search.mailList.flow'>原网站</a>以访问更多！
<br>
</div>
</div>
<!-- /. PAGE INNER -->
</div>
<!-- /. PAGE WRAPPER -->
</div>
<!-- /. WRAPPER -->
<!-- SCRIPTS -AT THE BOTOM TO REDUCE THE LOAD TIME-->
<!-- JQUERY SCRIPTS -->
<script src="../assets/js/jquery-1.10.2.js"></script>
<!-- BOOTSTRAP SCRIPTS -->
<script src="../assets/js/bootstrap.min.js"></script>
<!-- METISMENU SCRIPTS -->
<script src="../assets/js/jquery.metisMenu.js"></script>
<!-- CUSTOM SCRIPTS -->
<script src="../assets/js/custom.js"></script>
</body>
</html>
index.jsp
jsFiles
function makePageToPart1()
{
var Area = '';
Area += '<div class="row">';
Area += '<div class="col-md-12">';
Area += '<h2>每年信件数量</h2>';
Area += '</div>';
Area += '</div>';
Area += '<hr />';
Area += '<br>';
Area += '<div id="MessageArea">';
Area += ' <div id="displayArea" style="width:1000px;height:500px;"></div>';
Area += '</div>';
document.getElementById("page-inner").innerHTML = Area;
loadPart1();
}
function loadPart1()
{
var xmlHttp = null;
try{
xmlHttp = new XMLHttpRequest();
} catch (e1) {
try {
xmlHttp = new ActiveXObject("Microsoft.XMLHTTP");
} catch (e2) {
alert("Your browser does not support XMLHTTP!");
return;
}
}
xmlHttp.onreadystatechange = function() {
if (xmlHttp.readyState == 4) {
if (xmlHttp.status == 200)
{
s = xmlHttp.responseText;
var InformationSet = eval('('+s+')');
var leng = InformationSet;
var myChart = echarts.init(document.getElementById('displayArea'));
// 指定图表的配置项和数据
var option = {
title: {
subtext: '单位:条'
},
xAxis: {
name: '年份',
type: 'category',
boundaryGap: false,
axisLine: {onZero: false},
data: InformationSet.YEAR
},
yAxis: {
name: '数量',
type: 'value'
},
tooltip: {
trigger: 'axis'
},
series: [{
name: '数量',
data: InformationSet.NUM,
type: 'line',
markPoint: {
data: [
{type: 'max', name: '最大值'},
{type: 'min', name: '最小值'}
]
}
}]
};
// 使用刚指定的配置项和数据显示图表。
myChart.setOption(option);
}
}
};
var url ="../com/servlet/ServletForEveryYear"; xmlHttp.open("POST", url, true);
xmlHttp.setRequestHeader("Content-Type","application/x-www-form-urlencoded");
xmlHttp.send(null);
}
part1.js
function makePageToPart2()
{
var Area = '';
Area += '<div class="row">';
Area += '<div class="col-md-12">';
Area += '<h2>每种类型的信件数量</h2>';
Area += '</div>';
Area += '</div>';
Area += '<hr />';
Area += '<br>';
Area += '<div id="MessageArea">';
Area += ' <div id="displayArea" style="width:1000px;height:500px;"></div>';
Area += '</div>';
document.getElementById("page-inner").innerHTML = Area;
loadPart2();
}
function loadPart2()
{
var xmlHttp = null;
try{
xmlHttp = new XMLHttpRequest();
} catch (e1) {
try {
xmlHttp = new ActiveXObject("Microsoft.XMLHTTP");
} catch (e2) {
alert("Your browser does not support XMLHTTP!");
return;
}
}
xmlHttp.onreadystatechange = function() {
if (xmlHttp.readyState == 4) {
if (xmlHttp.status == 200)
{
s = xmlHttp.responseText;
var InformationSet = eval('('+s+')');
var myChart = echarts.init(document.getElementById('displayArea'));
var option = {
title: {
text: '每种类型的信件数量',
subtext: '数量',
left: 'center'
},
tooltip: {
trigger: 'item',
formatter: '数量 <br/>{b} : {c} ({d}%)'
},
legend: {
orient: 'vertical',
left: 'left',
data: InformationSet.KIND
},
series: [
{
name: '访问来源',
type: 'pie',
radius: '55%',
center: ['50%', '60%'],
data: InformationSet.DATA,
emphasis: {
itemStyle: {
shadowBlur: 10,
shadowOffsetX: 0,
shadowColor: 'rgba(0, 0, 0, 0.5)'
}
}
}
]
};
// 使用刚指定的配置项和数据显示图表。
myChart.setOption(option);
}
}
};
var url ="../com/servlet/ServletForKindData"; xmlHttp.open("POST", url, true);
xmlHttp.setRequestHeader("Content-Type","application/x-www-form-urlencoded");
xmlHttp.send(null);
}
part2.js
function makePageToPart3()
{
var Area = '';
Area += '<div class="row">';
Area += '<div class="col-md-12">';
Area += '<h2>每个政府部门处理的信件数量</h2>';
Area += '</div>';
Area += '</div>';
Area += '<hr />';
Area += '<br>';
Area += '<div id="MessageArea">';
Area += ' <div id="displayArea" style="width:1200px;height:500px;"></div>';
Area += '</div>';
document.getElementById("page-inner").innerHTML = Area;
loadPart3();
}
function loadPart3()
{
var xmlHttp = null;
try{
xmlHttp = new XMLHttpRequest();
} catch (e1) {
try {
xmlHttp = new ActiveXObject("Microsoft.XMLHTTP");
} catch (e2) {
alert("Your browser does not support XMLHTTP!");
return;
}
}
xmlHttp.onreadystatechange = function() {
if (xmlHttp.readyState == 4) {
if (xmlHttp.status == 200)
{
s = xmlHttp.responseText;
var InformationSet = eval('('+s+')');
var myChart = echarts.init(document.getElementById('displayArea'));
var option = {
xAxis: {
type: 'category',
data: InformationSet.RESPONSER
},
yAxis: {
type: 'value'
},
series: [{
data: InformationSet.NUM,
type: 'bar'
}]
};
// 使用刚指定的配置项和数据显示图表。
myChart.setOption(option);
}
}
};
var url ="../com/servlet/ServletForResponser"; xmlHttp.open("POST", url, true);
xmlHttp.setRequestHeader("Content-Type","application/x-www-form-urlencoded");
xmlHttp.send(null);
}
part3.js
接下来是页面展示：
----[ 主页面 ]
----[ 每年信件数量 ]
----[ 每种类型的信件数量]
----[ 每个政府部门处理的信件数量 ]
Python 爬取 北京市政府首都之窗信件列表-[信息展示]的更多相关文章
- Python 爬取 北京市政府首都之窗信件列表-[Scrapy框架](2020年寒假小目标04)
日期:2020.01.22 博客期:130 星期三 [代码说明,如果要使用此页代码,必须在本博客页面评论区给予说明] //博客总体说明 1.准备工作(本期博客) 2.爬取工作 3.数据处理 4.信息展 ...
- Python 爬取 北京市政府首都之窗信件列表-[后续补充]
日期:2020.01.23 博客期:131 星期四 [本博客的代码如若要使用,请在下方评论区留言,之后再用(就是跟我说一声)] //博客总体说明 1.准备工作 2.爬取工作(本期博客) 3.数据处理 ...
- Python 爬取 北京市政府首都之窗信件列表-[数据处理]
日期:2020.01.24 博客期:132 星期五 [代码说明,如果要使用此页代码,必须在本博客页面评论区给予说明] //博客总体说明 1.准备工作 2.爬取工作 3.数据处理(本期博客) 4.信息展 ...
- Python爬取网址中多个页面的信息
通过上一篇博客了解到爬取数据的操作,但对于存在多个页面的网址来说,使用上一篇博客中的代码爬取下来的资料并不完整.接下来就是讲解该如何爬取之后的页面信息. 一.审查元素 鼠标移至页码处右键,选择检查元素 ...
- python 爬取国家粮食局东北地区玉米收购价格监测信息
#!/usr/bin/python# -*- coding: UTF-8 -*-import reimport sysimport timeimport urllibimport urllib.req ...
- python爬取北京政府信件信息01
python爬取,找到目标地址,开始研究网页代码格式,于是就开始根据之前学的知识进行爬取,出师不利啊,一开始爬取就出现了个问题,这是之前是没有遇到过的,明明地址没问题,就是显示网页不存在,于是就在百度 ...
- Python爬取招聘信息，并且存储到MySQL数据库中
前面一篇文章主要讲述,如何通过Python爬取招聘信息,且爬取的日期为前一天的,同时将爬取的内容保存到数据库中:这篇文章主要讲述如何将python文件压缩成exe可执行文件,供后面的操作. 这系列文章 ...
- Python 爬取所有51VOA网站的Learn a words文本及mp3音频
Python 爬取所有51VOA网站的Learn a words文本及mp3音频 #!/usr/bin/env python # -*- coding: utf-8 -*- #Python 爬取所有5 ...
- python爬取网站数据
开学前接了一个任务,内容是从网上爬取特定属性的数据.正好之前学了python,练练手. 编码问题 因为涉及到中文,所以必然地涉及到了编码的问题,这一次借这个机会算是彻底搞清楚了. 问题要从文字的编码讲 ...
随机推荐
- IIS-简介
参考:https://www.jb51.net/article/85909.htm IIS是什么 iis是用来做什么的? IIS全程为Internet Information Service(In ...
- MYSQL 传汉字获取拼音首字母
--获取单个汉字首字母拼音 --CREATE DEFINER=`by`@`%` FUNCTION `fun_first_pinyin`(`P_NAME` VARCHAR(5)) RETURNS var ...
- 初始Linux软硬链接
转载 https://blog.csdn.net/libaineu2004/article/details/85918682 用实践让你明白"软链接"和"硬链接" ...
- 数据库中的null，踩坑笔记
问题来源:查询表中,complete_type 不为2的数据. 我的想法:select * from 表名 where complete_type != '2' (除了第2条,全部显示,没问题呀) ...
- 【PAT甲级】1074 Reversing Linked List (25 分)
题意: 输入链表头结点的地址(五位的字符串)和两个正整数N和K(N<=100000,K<=N),接着输入N行数据,每行包括结点的地址,结点的数据和下一个结点的地址.输出每K个结点局部反转的 ...
- Java面向对象private
1. 类 package cn.itcast.day06.demo03; /* 对于基本类型当中的boolean值,Getter方法一定要写成isXxx的形式,而setXxx规则不变. */ publ ...
- python应用-pycharm新建模板默认添加shebang编码作者时间等信息
1.pycharm4.5激活码 用户名: yueting3527 注册码: ===== LICENSE BEGIN ===== 93347-12042010 00001FMHemWIs"6w ...
- 吴裕雄--天生自然TensorFlow2教程：激活函数及其梯度
import tensorflow as tf a = tf.linspace(-10., 10., 10) a with tf.GradientTape() as tape: tape.watch( ...
- RTU：EvalRightToUse License for feature adventerprise 1.0 will transition to RightToUse in 10 days. UDI ASR1002-X：JAE2100XXXX
关于这个log:[Hostname] EvalRightToUse License for feature adventerprise 1.0 will transition to RightToUs ...
- IEEE 802.11r-2008
IEEE 802.11r-2008 or fast BSS transition (FT), also called fast roaming, is an amendment to the IEEE ...