【猫狗数据集】使用top1和top5准确率衡量模型
数据集下载地址:
链接:https://pan.baidu.com/s/1l1AnBgkAAEhh0vI5_loWKw
提取码:2xq4
创建数据集:https://www.cnblogs.com/xiximayou/p/12398285.html
读取数据集:https://www.cnblogs.com/xiximayou/p/12422827.html
进行训练:https://www.cnblogs.com/xiximayou/p/12448300.html
保存模型并继续进行训练:https://www.cnblogs.com/xiximayou/p/12452624.html
加载保存的模型并测试:https://www.cnblogs.com/xiximayou/p/12459499.html
划分验证集并边训练边验证:https://www.cnblogs.com/xiximayou/p/12464738.html
使用学习率衰减策略并边训练边测试:https://www.cnblogs.com/xiximayou/p/12468010.html
利用tensorboard可视化训练和测试过程:https://www.cnblogs.com/xiximayou/p/12482573.html
从命令行接收参数:https://www.cnblogs.com/xiximayou/p/12488662.html
epoch、batchsize、step之间的关系:https://www.cnblogs.com/xiximayou/p/12405485.html
之前使用的仅仅是top1准确率。在图像分类中,一般使用top1和top5来衡量分类模型的好坏。下面来看看。
首先在util下新建一个acc.py文件,向里面加入计算top1和top5准确率的代码:
import torch
def accu(output, target, topk=(1,)):
"""Computes the accuracy over the k top predictions for the specified values of k"""
with torch.no_grad():
maxk = max(topk)
batch_size = target.size(0)
_, pred = output.topk(maxk, 1, True, True)
pred = pred.t()
correct = pred.eq(target.view(1, -1).expand_as(pred)) res = []
for k in topk:
correct_k = correct[:k].view(-1).float().sum(0, keepdim=True)
res.append(correct_k.mul_(100.0 / batch_size))
return res
重点就是topk()函数:
torch.topk(input, k, dim=None, largest=True, sorted=True, out=None) -> (Tensor, LongTensor)
input:输入张量
k:指定返回的前几位的值
dim:排序的维度
largest:返回最大值
sorted:返回值是否排序
out:可选输出张量
需要注意的是我们这里只有两类,因此不存在top5。因此如果设置参数topk=(1,5),则会报错:RuntimeError:invalid argument 5:k not in range for dimension at /pytorch/ate ...
因此我们只能设置topk=(1,2),而且top2的值肯定是100%。最终res中第一位存储的是top1准确率,第二位存储的是top2准确率。
然后修改对应的train.py:
import torch
from tqdm import tqdm
from tensorflow import summary
import datetime
from utils import acc """
current_time = str(datetime.datetime.now().timestamp())
train_log_dir = '/content/drive/My Drive/colab notebooks/output/tsboardx/train/' + current_time
test_log_dir = '/content/drive/My Drive/colab notebooks/output/tsboardx/test/' + current_time
val_log_dir = '/content/drive/My Drive/colab notebooks/output/tsboardx/val/' + current_time
train_summary_writer = summary.create_file_writer(train_log_dir)
val_summary_writer = summary.create_file_writer(val_log_dir)
test_summary_writer = summary.create_file_writer(test_log_dir)
"""
class Trainer:
def __init__(self,criterion,optimizer,model):
self.criterion=criterion
self.optimizer=optimizer
self.model=model
def get_lr(self):
for param_group in self.optimizer.param_groups:
return param_group['lr']
def loop(self,num_epochs,train_loader,val_loader,test_loader,scheduler=None,acc1=0.0):
self.acc1=acc1
for epoch in range(1,num_epochs+1):
lr=self.get_lr()
print("epoch:{},lr:{:.6f}".format(epoch,lr))
self.train(train_loader,epoch,num_epochs)
self.val(val_loader,epoch,num_epochs)
self.test(test_loader,epoch,num_epochs)
if scheduler is not None:
scheduler.step() def train(self,dataloader,epoch,num_epochs):
self.model.train()
with torch.enable_grad():
self._iteration_train(dataloader,epoch,num_epochs) def val(self,dataloader,epoch,num_epochs):
self.model.eval()
with torch.no_grad():
self._iteration_val(dataloader,epoch,num_epochs)
def test(self,dataloader,epoch,num_epochs):
self.model.eval()
with torch.no_grad():
self._iteration_test(dataloader,epoch,num_epochs) def _iteration_train(self,dataloader,epoch,num_epochs):
#total_step=len(dataloader)
#tot_loss = 0.0
#correct = 0
train_loss=AverageMeter()
train_top1=AverageMeter()
train_top2=AverageMeter()
#for i ,(images, labels) in enumerate(dataloader):
#res=[]
for images, labels in tqdm(dataloader,ncols=80):
images = images.cuda()
labels = labels.cuda()
# Forward pass
outputs = self.model(images)
#_, preds = torch.max(outputs.data,1)
pred1_train,pred2_train=acc.accu(outputs,labels,topk=(1,2))
loss = self.criterion(outputs, labels)
train_loss.update(loss.item(),images.size(0))
train_top1.update(pred1_train[0],images.size(0))
train_top2.update(pred2_train[0],images.size(0))
# Backward and optimizer
self.optimizer.zero_grad()
loss.backward()
self.optimizer.step()
#tot_loss += loss.data
"""
if (i+1) % 2 == 0:
print('Epoch: [{}/{}], Step: [{}/{}], Loss: {:.4f}'
.format(epoch, num_epochs, i+1, total_step, loss.item()))
"""
#correct += torch.sum(preds == labels.data).to(torch.float32)
### Epoch info ####
#epoch_loss = tot_loss/len(dataloader.dataset)
#epoch_acc = correct/len(dataloader.dataset)
#print('train loss: {:.4f},train acc: {:.4f}'.format(epoch_loss,epoch_acc))
print(">>>[{}] train_loss:{:.4f} top1:{:.4f} top2:{:.4f}".format("train", train_loss.avg, train_top1.avg, train_top2.avg))
"""
with train_summary_writer.as_default():
summary.scalar('loss', train_loss.avg, epoch)
summary.scalar('accuracy', train_top1.avg, epoch)
"""
"""
if epoch==num_epochs:
state = {
'model': self.model.state_dict(),
'optimizer':self.optimizer.state_dict(),
'epoch': epoch,
'train_loss':train_loss.avg,
'train_acc':train_top1.avg,
}
save_path="/content/drive/My Drive/colab notebooks/output/"
torch.save(state,save_path+"/resnet18_final_v2"+".t7")
"""
t_loss = train_loss.avg,
t_top1 = train_top1.avg
t_top2 = train_top2.avg
return t_loss,t_top1,t_top2
def _iteration_val(self,dataloader,epoch,num_epochs):
#total_step=len(dataloader)
#tot_loss = 0.0
#correct = 0
#for i ,(images, labels) in enumerate(dataloader):
val_loss=AverageMeter()
val_top1=AverageMeter()
val_top2=AverageMeter()
for images, labels in tqdm(dataloader,ncols=80):
images = images.cuda()
labels = labels.cuda() # Forward pass
outputs = self.model(images)
#_, preds = torch.max(outputs.data,1)
pred1_val,pred2_val=acc.accu(outputs,labels,topk=(1,2))
loss = self.criterion(outputs, labels)
val_loss.update(loss.item(),images.size(0))
val_top1.update(pred1_val[0],images.size(0))
val_top2.update(pred2_val[0],images.size(0))
#tot_loss += loss.data
#correct += torch.sum(preds == labels.data).to(torch.float32)
"""
if (i+1) % 2 == 0:
print('Epoch: [{}/{}], Step: [{}/{}], Loss: {:.4f}'
.format(1, 1, i+1, total_step, loss.item()))
"""
### Epoch info ####
#epoch_loss = tot_loss/len(dataloader.dataset)
#epoch_acc = correct/len(dataloader.dataset)
#print('val loss: {:.4f},val acc: {:.4f}'.format(epoch_loss,epoch_acc))
print(">>>[{}] val_loss:{:.4f} top1:{:.4f} top2:{:.4f}".format("val", val_loss.avg, val_top1.avg, val_top2.avg))
"""
with val_summary_writer.as_default():
summary.scalar('loss', val_loss.avg, epoch)
summary.scalar('accuracy', val_top1.avg, epoch)
"""
t_loss = val_loss.avg,
t_top1 = val_top1.avg
t_top2 = val_top2.avg
return t_loss,t_top1,t_top2
def _iteration_test(self,dataloader,epoch,num_epochs):
#total_step=len(dataloader)
#tot_loss = 0.0
#correct = 0
#for i ,(images, labels) in enumerate(dataloader):
test_loss=AverageMeter()
test_top1=AverageMeter()
test_top2=AverageMeter()
for images, labels in tqdm(dataloader,ncols=80):
images = images.cuda()
labels = labels.cuda() # Forward pass
outputs = self.model(images)
#_, preds = torch.max(outputs.data,1)
pred1_test,pred2_test=acc.accu(outputs,labels,topk=(1,2))
loss = self.criterion(outputs, labels)
test_loss.update(loss.item(),images.size(0))
test_top1.update(pred1_test[0],images.size(0))
test_top2.update(pred2_test[0],images.size(0))
#tot_loss += loss.data
#correct += torch.sum(preds == labels.data).to(torch.float32)
"""
if (i+1) % 2 == 0:
print('Epoch: [{}/{}], Step: [{}/{}], Loss: {:.4f}'
.format(1, 1, i+1, total_step, loss.item()))
"""
### Epoch info ####
#epoch_loss = tot_loss/len(dataloader.dataset)
#epoch_acc = correct/len(dataloader.dataset)
#print('test loss: {:.4f},test acc: {:.4f}'.format(epoch_loss,epoch_acc))
print(">>>[{}] test_loss:{:.4f} top1:{:.4f} top2:{:.4f}".format("test", test_loss.avg, test_top1.avg, test_top2.avg))
t_loss = test_loss.avg,
t_top1 = test_top1.avg
t_top2 = test_top2.avg
"""
with test_summary_writer.as_default():
summary.scalar('loss', test_loss.avg, epoch)
summary.scalar('accuracy', test_top1.avg, epoch)
"""
"""
if epoch_acc > self.acc1:
state = {
"model": self.model.state_dict(),
"optimizer": self.optimizer.state_dict(),
"epoch": epoch,
"epoch_loss": test_loss.avg,
"epoch_acc": test_top1.avg,
}
save_path="/content/drive/My Drive/colab notebooks/output/"
print("在第{}个epoch取得最好的测试准确率,准确率为:{:.4f}".format(epoch,test_loss.avg))
torch.save(state,save_path+"/resnet18_best_v2"+".t7")
self.acc1=max(self.acc1,test_loss.avg)
"""
return t_loss,t_top1,t_top2 class AverageMeter(object):
def __init__(self):
self.reset() def reset(self):
self.val = 0
self.avg = 0
self.sum = 0
self.count = 0 def update(self, val, n=1):
self.val = val
self.sum += float(val) * n
self.count += n
self.avg = self.sum / self.count
我们新建了一个AverageMeter类来存储结果。
最终结果:
下一节:加载预训练的模型并进行微调。
【猫狗数据集】使用top1和top5准确率衡量模型的更多相关文章
- 【猫狗数据集】使用预训练的resnet18模型
数据集下载地址: 链接:https://pan.baidu.com/s/1l1AnBgkAAEhh0vI5_loWKw提取码:2xq4 创建数据集:https://www.cnblogs.com/xi ...
- 【猫狗数据集】pytorch训练猫狗数据集之创建数据集
猫狗数据集的分为训练集25000张,在训练集中猫和狗的图像是混在一起的,pytorch读取数据集有两种方式,第一种方式是将不同类别的图片放于其对应的类文件夹中,另一种是实现读取数据集类,该类继承tor ...
- 【猫狗数据集】利用tensorboard可视化训练和测试过程
数据集下载地址: 链接:https://pan.baidu.com/s/1l1AnBgkAAEhh0vI5_loWKw提取码:2xq4 创建数据集:https://www.cnblogs.com/xi ...
- 【猫狗数据集】谷歌colab之使用pytorch读取自己数据集(猫狗数据集)
之前在:https://www.cnblogs.com/xiximayou/p/12398285.html创建好了数据集,将它上传到谷歌colab 在colab上的目录如下: 在utils中的rdat ...
- kaggle之猫狗数据集下载
链接:https://pan.baidu.com/s/1l1AnBgkAAEhh0vI5_loWKw 提取码:2xq4 百度网盘实在是恶心,找的别人的网盘下载不仅速度慢,还老挂掉,自己去kaggle下 ...
- Pytorch实现Top1准确率和Top5准确率
之前一直不清楚Top1和Top5是什么,其实搞清楚了很简单,就是两种衡量指标,其中,Top1就是普通的Accuracy,Top5比Top1衡量标准更“严格”, 具体来讲,比如一共需要分10类,每次分类 ...
- 深度学习原理与框架-猫狗图像识别-卷积神经网络(代码) 1.cv2.resize(图片压缩) 2..get_shape()[1:4].num_elements(获得最后三维度之和) 3.saver.save(训练参数的保存) 4.tf.train.import_meta_graph(加载模型结构) 5.saver.restore(训练参数载入)
1.cv2.resize(image, (image_size, image_size), 0, 0, cv2.INTER_LINEAR) 参数说明:image表示输入图片,image_size表示变 ...
- 使用pytorch完成kaggle猫狗图像识别
kaggle是一个为开发商和数据科学家提供举办机器学习竞赛.托管数据库.编写和分享代码的平台,在这上面有非常多的好项目.好资源可供机器学习.深度学习爱好者学习之用.碰巧最近入门了一门非常的深度学习框架 ...
- paddlepaddle实现猫狗分类
目录 1.预备工作 1.1 数据集准备 1.2 数据预处理 2.训练 2.1 模型 2.2 定义训练 2.3 训练 3.预测 4.参考文献 声明:这是我的个人学习笔记,大佬可以点评,指导,不喜勿喷.实 ...
随机推荐
- 树剖裸题——BZOJ1036 树的统计
#include<cstring> #include<cmath> #include<algorithm> #include<cstdio> #defi ...
- php配置memcached的扩展。
(一)安装memcached服务器 1根据系统下载相应版本的memcached服务器版本:如win7(64位=====>memcached-win64/memcached.exe 2.解压到目录 ...
- MOOC(7)- case依赖、读取json配置文件进行多个接口请求-跳过测试用例(6)
初始化.跳过测试用例 # test_class_6.py import unittest from mock import mock from day_20200208_mooc.base.inter ...
- oracle存储过程随笔
有时间要把最近总结的存储过程的问题总结一下,先占坑: create or replace procedure test_proc2 as cursor cur is select YWR,YWRZJH ...
- Dubbo+zookeeper 基础讲解
一.dubbo是什么? 1)本质:一个Jar包,一个分布式框架,,一个远程服务调用的分布式框架. 既然是新手教学,肯定很多同学不明白什么是分布式和远程服务调用,为什么要分布式,为什么要远程调用.我简单 ...
- 基于Docker 部署Jmeter + Grafana + InfluxDB 性能测试监控配置(亲测可用)
工具介绍: InfluxDB:是一款用Go语言编写的开源分布式时序.事件和指标数据库,无需外部依赖.该数据库现在主要用于存储涉及大量的时间戳数据,如DevOps监控数据,APP metrics, lo ...
- spring security之Remember Me
spring-security.xml配置 环境: spring版本:5.0.7.RELEASE spring-security.xml引入: http://www.springframework.o ...
- 写了个通作的分页存储过程,top,加入了排序
USE [WebDB_TradeOrder]GO/****** Object: StoredProcedure [dbo].[Boss_Proc_PagingWithOrder] Script ...
- JAVA递归、非递归遍历二叉树
前序遍历:1.访问根节点 2.前序遍历左子树 3.前序遍历右子树 中序遍历:1.中序遍历左子树 2.访问根节点 3.中序遍历右子树 后序遍历:1.后序遍历左子树 2.后序遍历右子树 3.访问根节点-- ...
- MySQL数据类型(DATA Type)与数据恢复与备份方法
一.数据类型(DATA Type)概述 MySQL支持多种类型的SQL数据类型:数字类型,日期和时间类型,字符串(字符和字节)类型以及空间类型 数据类型描述使用以下约定: M表示整数类型的最大显示宽度 ...