0%

机器学习实战 --（1）regression

發表於 2022-05-07 更新於 2022-05-08

一、前言

Objectives:

Solve a regression problem with deep neural networks (DNN).
Understand basic DNN training tips.
Familiarize yourself with PyTorch.

二、引入 python 库

"""# Import packages"""

# Numerical Operations
import math
import numpy as np

# Reading/Writing Data
import pandas as pd
import os
import csv

# For Progress Bar
from tqdm import tqdm

# Pytorch
import torch 
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader, random_split

# For plotting learning curve
from torch.utils.tensorboard import SummaryWriter

三、Utility Functions


def same_seed (seed): 
    '''Fixes random number generator seeds for reproducibility.'''
    torch.backends.cudnn.deterministic = True
    torch.backends.cudnn.benchmark = False
    np.random.seed (seed)
    torch.manual_seed (seed)
    if torch.cuda.is_available ():
        torch.cuda.manual_seed_all (seed)

def train_valid_split (data_set, valid_ratio, seed):
    '''Split provided training data into training set and validation set'''
    valid_set_size = int (valid_ratio * len (data_set)) 
    train_set_size = len (data_set) - valid_set_size
    train_set, valid_set = random_split (data_set, [train_set_size, valid_set_size], generator=torch.Generator ().manual_seed (seed))
    return np.array (train_set), np.array (valid_set)

def predict (test_loader, model, device):
    model.eval () # Set your model to evaluation mode.
    preds = []
    for x in tqdm (test_loader):
        x = x.to (device)                        
        with torch.no_grad ():                   
            pred = model (x)                     
            preds.append (pred.detach ().cpu ())   
    preds = torch.cat (preds, dim=0).numpy ()  
    return preds


 设置 torch.backends.cudnn.benchmark=True 将会让程序在开始时花费一点额外时间，为整个网络的每个卷积层搜索最适合它的卷积实现算法，进而实现网络的加速。适用场景是网络结构固定（不是动态变化的），网络的输入形状（包括 batch size，图片大小，输入的通道）是不变的，其实也就是一般情况下都比较适用。反之，如果卷积层的设置一直变化，将会导致程序不停地做优化，反而会耗费更多的时间。

torch.backends.cudnn.deterministic=True
 每次返回的卷积算法将是确定的，即默认算法。cudnn 中包含很多卷积算法。基于 GEMM (General Matrix Multiply) 的，基于 FFT 的，基于 Winograd 算法的等等。

 设置 seed () 里的数字就相当于设置了一个盛有随机数的 “聚宝盆”，一个数字代表一个 “聚宝盆”，当我们在 seed（）的括号里设置相同的 seed，“聚宝盆” 就是一样的，从里面拿出的随机数就会相同。

<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">rng</span> ():</span></span><br><span class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span> (<span class="number">5</span>):</span><br><span class="line">        np.random.seed (<span class="number">123</span>)</span><br><span class="line">        <span class="built_in">print</span> (np.random.rand (<span class="number">4</span>))</span><br><span class="line">        </span><br><span class="line">rng ()</span><br><span class="line">&gt;&gt;&gt;[<span class="number">0.69646919</span> <span class="number">0.28613933</span> <span class="number">0.22685145</span> <span class="number">0.55131477</span>]</span><br><span class="line">   [<span class="number">0.69646919</span> <span class="number">0.28613933</span> <span class="number">0.22685145</span> <span class="number">0.55131477</span>]</span><br><span class="line">   [<span class="number">0.69646919</span> <span class="number">0.28613933</span> <span class="number">0.22685145</span> <span class="number">0.55131477</span>]</span><br><span class="line">   [<span class="number">0.69646919</span> <span class="number">0.28613933</span> <span class="number">0.22685145</span> <span class="number">0.55131477</span>]</span><br><span class="line">   [<span class="number">0.69646919</span> <span class="number">0.28613933</span> <span class="number">0.22685145</span> <span class="number">0.55131477</span>]</span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">rng_n</span> ():</span></span><br><span class="line">    np.random.seed (<span class="number">123</span>)</span><br><span class="line">    <span class="keyword">for</span> i <span class="keyword">in</span> <span class="built_in">range</span> (<span class="number">5</span>):</span><br><span class="line">        <span class="built_in">print</span> (np.random.rand (<span class="number">4</span>))</span><br><span class="line">        </span><br><span class="line">rng_n ()</span><br><span class="line">&gt;&gt;&gt;[<span class="number">0.69646919</span> <span class="number">0.28613933</span> <span class="number">0.22685145</span> <span class="number">0.55131477</span>]</span><br><span class="line">   [<span class="number">0.71946897</span> <span class="number">0.42310646</span> <span class="number">0.9807642</span>  <span class="number">0.68482974</span>]</span><br><span class="line">   [<span class="number">0.4809319</span>  <span class="number">0.39211752</span> <span class="number">0.34317802</span> <span class="number">0.72904971</span>]</span><br><span class="line">   [<span class="number">0.43857224</span> <span class="number">0.0596779</span>  <span class="number">0.39804426</span> <span class="number">0.73799541</span>]</span><br><span class="line">   [<span class="number">0.18249173</span> <span class="number">0.17545176</span> <span class="number">0.53155137</span> <span class="number">0.53182759</span>]</span><br></pre></td></tr></table></figure>
<p> 在神经网络中，参数默认是进行随机初始化的。如果不设置的话每次训练时的初始化都是随机的，导致结果不确定。如果设置初始化，则每次初始化都是固定的。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br></pre></td><td class="code"><pre><span class="line"><span class="keyword">if</span> args.seed <span class="keyword">is</span> <span class="keyword">not</span> <span class="literal">None</span>:</span><br><span class="line">    random.seed (args.seed) <span class="comment">#</span></span><br><span class="line">    torch.manual_seed (args.seed)  <span class="comment"># 为 CPU 设置种子用于生成随机数，以使得结果是确定的   　　 torch.cuda.manual_seed (args.seed) #为当前 GPU 设置随机种子；</span></span><br><span class="line">    cudnn.deterministic = <span class="literal">True</span></span><br><span class="line"></span><br><span class="line"><span class="comment"># 如果使用多个 GPU，应该使用 torch.cuda.manual_seed_all () 为所有的 GPU 设置种子。</span></span><br></pre></td></tr></table></figure></p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br></pre></td><td class="code"><pre><span class="line">torch.utils.data.random_split (dataset, lengths, generator=&lt;torch._C.Generator <span class="built_in">object</span>&gt;)</span><br></pre></td></tr></table></figure>
<p> 在 CPU 中设置生成随机数的种子，并返回一个 torch.Generator 对象。当设置的种子固定下来的时候，之后依次 pytorch 生成的随机数序列也被固定下来。需要注意的是当只调用 torch.manual_seed () 一次时并不能生成相同的随机数序列。如果想要得到相同的随机数序列就需要每次产生随机数的时候都要调用一下 torch.manual_seed ()。<br><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br></pre></td><td class="code"><pre><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="keyword">import</span> torch</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span>torch.manual_seed (<span class="number">2</span>)</span><br><span class="line">&lt;torch._C.Generator <span class="built_in">object</span> at <span class="number">0x000001EB8F3A1918</span>&gt;</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="built_in">print</span> (torch.randn (<span class="number">2</span>,<span class="number">2</span>))</span><br><span class="line">tensor ([[<span class="number">0.3923</span>, -<span class="number">0.2236</span>],</span><br><span class="line">        [-<span class="number">0.3195</span>, -<span class="number">1.2050</span>]])</span><br><span class="line"><span class="meta">&gt;&gt;&gt; </span><span class="built_in">print</span> (torch.randn (<span class="number">2</span>,<span class="number">2</span>))</span><br><span class="line">tensor ([[<span class="number">1.0445</span>, -<span class="number">0.6332</span>],</span><br><span class="line">        [<span class="number">0.5731</span>,  <span class="number">0.5409</span>]])</span><br><span class="line"></span><br></pre></td></tr></table></figure></p>
<p><a href="https://zhuanlan.zhihu.com/p/357075502">Pytorch：model.train () 和 model.eval () 用法和区别 </a></p>
<h3 id=" 四、Dataset”"><a href="# 四、Dataset”" class="headerlink" title=" 四、Dataset”"></a> 四、Dataset”</h3><figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">COVID19Dataset</span> (<span class="params">Dataset</span>):</span></span><br><span class="line">    <span class="string">&#x27;&#x27;&#x27;</span></span><br><span class="line"><span class="string">    x: Features.</span></span><br><span class="line"><span class="string">    y: Targets, if none, do prediction.</span></span><br><span class="line"><span class="string">    &#x27;&#x27;&#x27;</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span>(<span class="params">self, x, y=<span class="literal">None</span></span>):</span></span><br><span class="line">        <span class="keyword">if</span> y <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line">            self.y = y</span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            self.y = torch.FloatTensor (y)</span><br><span class="line">        self.x = torch.FloatTensor (x)</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__getitem__</span>(<span class="params">self, idx</span>):</span></span><br><span class="line">        <span class="keyword">if</span> self.y <span class="keyword">is</span> <span class="literal">None</span>:</span><br><span class="line">            <span class="keyword">return</span> self.x [idx]</span><br><span class="line">        <span class="keyword">else</span>:</span><br><span class="line">            <span class="keyword">return</span> self.x [idx], self.y [idx]</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__len__</span>(<span class="params">self</span>):</span></span><br><span class="line">        <span class="keyword">return</span> <span class="built_in">len</span> (self.x)</span><br></pre></td></tr></table></figure>
<h3 id=" 五、Neural-Network-Model"><a href="# 五、Neural-Network-Model" class="headerlink" title=" 五、Neural Network Model"></a> 五、Neural Network Model</h3><p>Try out different model architectures by modifying the class below.</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br></pre></td><td class="code"><pre><span class="line"><span class="class"><span class="keyword">class</span> <span class="title">My_Model</span> (<span class="params">nn.Module</span>):</span></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">__init__</span>(<span class="params">self, input_dim</span>):</span></span><br><span class="line">        <span class="built_in">super</span> (My_Model, self).__init__()</span><br><span class="line">        <span class="comment"># <span class="doctag">TODO:</span> modify model&#x27;s structure, be aware of dimensions. </span></span><br><span class="line">        self.layers = nn.Sequential (</span><br><span class="line">            nn.Linear (input_dim, <span class="number">16</span>),</span><br><span class="line">            nn.ReLU (),</span><br><span class="line">            nn.Linear (<span class="number">16</span>, <span class="number">8</span>),</span><br><span class="line">            nn.ReLU (),</span><br><span class="line">            nn.Linear (<span class="number">8</span>, <span class="number">1</span>)</span><br><span class="line">        )</span><br><span class="line"></span><br><span class="line">    <span class="function"><span class="keyword">def</span> <span class="title">forward</span> (<span class="params">self, x</span>):</span></span><br><span class="line">        x = self.layers (x)</span><br><span class="line">        x = x.squeeze (<span class="number">1</span>) <span class="comment"># (B, 1) -&gt; (B)</span></span><br><span class="line">        <span class="keyword">return</span> x</span><br></pre></td></tr></table></figure>
<p>Pytorch Tensor 的通道排序：[batch, channel, height, weight]<br><a href="https://pytorch.org/tutorials/beginner/blitz/neural_networks_tutorial.html">pytorch lenet</a><br><a href="https://blog.csdn.net/jokerxsy/article/details/108614661">nn.Conv1d\nn.Conv2d 以及 groups\dilation 参数的理解 </a><br><a href="https://www.jianshu.com/p/c1232e47661f">PyTorch 教程 - 3：PyTorch 中神经网络的构建与训练基础 </a><br><a href="https://blog.csdn.net/qq_26442553/article/details/81775449">Python 多继承与 super 使用详解 </a></p>
<p><img src="./machineLearning-1/1.png" alt=""></p>
<p><a href="https://blog.csdn.net/york1996/article/details/81949843">PyTorch 中 view 的用法 </a><br><a href="https://zhuanlan.zhihu.com/p/27382990"> 图像基本操作 torchvision.transforms</a></p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">main</span> ():</span></span><br><span class="line">    transform = transforms.Compose (</span><br><span class="line">        [transforms.ToTensor (),</span><br><span class="line">         transforms.Normalize ((<span class="number">0.5</span>, <span class="number">0.5</span>, <span class="number">0.5</span>), (<span class="number">0.5</span>, <span class="number">0.5</span>, <span class="number">0.5</span>))])</span><br><span class="line"></span><br><span class="line">    <span class="comment"># 50000 张训练图片 </span></span><br><span class="line">    <span class="comment"># 第一次使用时要将 download 设置为 True 才会自动去下载数据集 </span></span><br><span class="line">    train_set = torchvision.datasets.CIFAR10 (root=<span class="string">&#x27;./data&#x27;</span>, train=<span class="literal">True</span>,</span><br><span class="line">                                             download=<span class="literal">False</span>, transform=transform)</span><br><span class="line">    train_loader = torch.utils.data.DataLoader (train_set, batch_size=<span class="number">36</span>,</span><br><span class="line">                                               shuffle=<span class="literal">True</span>, num_workers=<span class="number">0</span>)</span><br><span class="line"></span><br><span class="line">    <span class="comment"># 10000 张验证图片 </span></span><br><span class="line">    <span class="comment"># 第一次使用时要将 download 设置为 True 才会自动去下载数据集 </span></span><br><span class="line">    val_set = torchvision.datasets.CIFAR10 (root=<span class="string">&#x27;./data&#x27;</span>, train=<span class="literal">False</span>,</span><br><span class="line">                                           download=<span class="literal">False</span>, transform=transform)</span><br><span class="line">    val_loader = torch.utils.data.DataLoader (val_set, batch_size=<span class="number">5000</span>,</span><br><span class="line">                                             shuffle=<span class="literal">False</span>, num_workers=<span class="number">0</span>)</span><br><span class="line">    val_data_iter = <span class="built_in">iter</span> (val_loader)</span><br><span class="line">    val_image, val_label = val_data_iter.<span class="built_in">next</span> ()</span><br><span class="line">    ...</span><br></pre></td></tr></table></figure>
<p><img src="./machineLearning-1/photo2.png" alt=""></p>
<h3 id=" 六、Feature-Selection"><a href="# 六、Feature-Selection" class="headerlink" title=" 六、Feature Selection"></a> 六、Feature Selection</h3><p>Choose features you deem useful by modifying the function below.</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br></pre></td><td class="code"><pre><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">select_feat</span> (<span class="params">train_data, valid_data, test_data, select_all=<span class="literal">True</span></span>):</span></span><br><span class="line">    <span class="string">&#x27;&#x27;&#x27;Selects useful features to perform regression&#x27;&#x27;&#x27;</span></span><br><span class="line">    y_train, y_valid = train_data [:,-<span class="number">1</span>], valid_data [:,-<span class="number">1</span>]</span><br><span class="line">    raw_x_train, raw_x_valid, raw_x_test = train_data [:,:-<span class="number">1</span>], valid_data [:,:-<span class="number">1</span>], test_data</span><br><span class="line"></span><br><span class="line">    <span class="keyword">if</span> select_all:</span><br><span class="line">        feat_idx = <span class="built_in">list</span> (<span class="built_in">range</span> (raw_x_train.shape [<span class="number">1</span>]))</span><br><span class="line">    <span class="keyword">else</span>:</span><br><span class="line">        feat_idx = [<span class="number">0</span>,<span class="number">1</span>,<span class="number">2</span>,<span class="number">3</span>,<span class="number">4</span>] <span class="comment"># <span class="doctag">TODO:</span> Select suitable feature columns.</span></span><br><span class="line">        </span><br><span class="line">    <span class="keyword">return</span> raw_x_train [:,feat_idx], raw_x_valid [:,feat_idx], raw_x_test [:,feat_idx], y_train, y_valid</span><br></pre></td></tr></table></figure>
<h3 id=" 六、Training-Loop"><a href="# 六、Training-Loop" class="headerlink" title=" 六、Training Loop"></a> 六、Training Loop</h3><p>(1) batchsize：批大小。在深度学习中，一般采用 SGD 训练，即每次训练在训练集中取 batchsize 个样本训练；<br>(2) iteration：1 个 iteration 等于使用 batchsize 个样本训练一次；<br>(3) epoch：1 个 epoch 等于使用训练集中的全部样本训练一次；</p>
<p> 举个例子，训练集有 1000 个样本，batchsize=10，那么，<br> 训练完整个样本集需要：100 次 iteration，1 次 epoch。</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br><span class="line">13</span><br><span class="line">14</span><br><span class="line">15</span><br><span class="line">16</span><br><span class="line">17</span><br><span class="line">18</span><br><span class="line">19</span><br><span class="line">20</span><br><span class="line">21</span><br><span class="line">22</span><br><span class="line">23</span><br><span class="line">24</span><br><span class="line">25</span><br><span class="line">26</span><br><span class="line">27</span><br><span class="line">28</span><br><span class="line">29</span><br><span class="line">30</span><br><span class="line">31</span><br><span class="line">32</span><br><span class="line">33</span><br><span class="line">34</span><br><span class="line">35</span><br><span class="line">36</span><br><span class="line">37</span><br><span class="line">38</span><br><span class="line">39</span><br><span class="line">40</span><br><span class="line">41</span><br><span class="line">42</span><br><span class="line">43</span><br><span class="line">44</span><br><span class="line">45</span><br><span class="line">46</span><br><span class="line">47</span><br><span class="line">48</span><br><span class="line">49</span><br><span class="line">50</span><br><span class="line">51</span><br><span class="line">52</span><br><span class="line">53</span><br><span class="line">54</span><br><span class="line">55</span><br><span class="line">56</span><br><span class="line">57</span><br><span class="line">58</span><br><span class="line">59</span><br><span class="line">60</span><br><span class="line">61</span><br><span class="line">62</span><br><span class="line">63</span><br><span class="line">64</span><br><span class="line">65</span><br><span class="line">66</span><br></pre></td><td class="code"><pre><span class="line"></span><br><span class="line"><span class="function"><span class="keyword">def</span> <span class="title">trainer</span> (<span class="params">train_loader, valid_loader, model, config, device</span>):</span></span><br><span class="line"></span><br><span class="line">    criterion = nn.MSELoss (reduction=<span class="string">&#x27;mean&#x27;</span>) <span class="comment"># Define your loss function, do not modify this.</span></span><br><span class="line"></span><br><span class="line">    <span class="comment"># Define your optimization algorithm. </span></span><br><span class="line">    <span class="comment"># <span class="doctag">TODO:</span> Please check https://pytorch.org/docs/stable/optim.html to get more available algorithms.</span></span><br><span class="line">    <span class="comment"># <span class="doctag">TODO:</span> L2 regularization (optimizer (weight decay...) or implement by your self).</span></span><br><span class="line">    optimizer = torch.optim.SGD (model.parameters (), lr=config [<span class="string">&#x27;learning_rate&#x27;</span>], momentum=<span class="number">0.9</span>) </span><br><span class="line"></span><br><span class="line">    writer = SummaryWriter () <span class="comment"># Writer of tensoboard.</span></span><br><span class="line"></span><br><span class="line">    <span class="keyword">if</span> <span class="keyword">not</span> os.path.isdir (<span class="string">&#x27;./models&#x27;</span>):</span><br><span class="line">        os.mkdir (<span class="string">&#x27;./models&#x27;</span>) <span class="comment"># Create directory of saving models.</span></span><br><span class="line"></span><br><span class="line">    n_epochs, best_loss, step, early_stop_count = config [<span class="string">&#x27;n_epochs&#x27;</span>], math.inf, <span class="number">0</span>, <span class="number">0</span></span><br><span class="line"></span><br><span class="line">    <span class="keyword">for</span> epoch <span class="keyword">in</span> <span class="built_in">range</span> (n_epochs):</span><br><span class="line">        model.train () <span class="comment"># Set your model to train mode.</span></span><br><span class="line">        loss_record = []</span><br><span class="line"></span><br><span class="line">        <span class="comment"># tqdm is a package to visualize your training progress.</span></span><br><span class="line">        train_pbar = tqdm (train_loader, position=<span class="number">0</span>, leave=<span class="literal">True</span>)</span><br><span class="line"></span><br><span class="line">        <span class="keyword">for</span> x, y <span class="keyword">in</span> train_pbar:</span><br><span class="line">            optimizer.zero_grad ()               <span class="comment"># Set gradient to zero.</span></span><br><span class="line">            x, y = x.to (device), y.to (device)   <span class="comment"># Move your data to device. </span></span><br><span class="line">            pred = model (x)             </span><br><span class="line">            loss = criterion (pred, y)</span><br><span class="line">            loss.backward ()                     <span class="comment"># Compute gradient (backpropagation).</span></span><br><span class="line">            optimizer.step ()                    <span class="comment"># Update parameters.</span></span><br><span class="line">            step += <span class="number">1</span></span><br><span class="line">            loss_record.append (loss.detach ().item ())</span><br><span class="line">            </span><br><span class="line">            <span class="comment"># Display current epoch number and loss on tqdm progress bar.</span></span><br><span class="line">            train_pbar.set_description (<span class="string">f&#x27;Epoch [<span class="subst">&#123;epoch+<span class="number">1</span>&#125;</span>/<span class="subst">&#123;n_epochs&#125;</span>]&#x27;</span>)</span><br><span class="line">            train_pbar.set_postfix (&#123;<span class="string">&#x27;loss&#x27;</span>: loss.detach ().item ()&#125;)</span><br><span class="line"></span><br><span class="line">        mean_train_loss = <span class="built_in">sum</span> (loss_record)/<span class="built_in">len</span> (loss_record)</span><br><span class="line">        writer.add_scalar (<span class="string">&#x27;Loss/train&#x27;</span>, mean_train_loss, step)</span><br><span class="line"></span><br><span class="line">        model.<span class="built_in">eval</span> () <span class="comment"># Set your model to evaluation mode.</span></span><br><span class="line">        loss_record = []</span><br><span class="line">        <span class="keyword">for</span> x, y <span class="keyword">in</span> valid_loader:</span><br><span class="line">            x, y = x.to (device), y.to (device)</span><br><span class="line">            <span class="keyword">with</span> torch.no_grad ():</span><br><span class="line">                pred = model (x)</span><br><span class="line">                loss = criterion (pred, y)</span><br><span class="line"></span><br><span class="line">            loss_record.append (loss.item ())</span><br><span class="line">            </span><br><span class="line">        mean_valid_loss = <span class="built_in">sum</span> (loss_record)/<span class="built_in">len</span> (loss_record)</span><br><span class="line">        <span class="built_in">print</span> (<span class="string">f&#x27;Epoch [<span class="subst">&#123;epoch+<span class="number">1</span>&#125;</span>/<span class="subst">&#123;n_epochs&#125;</span>]: Train loss: <span class="subst">&#123;mean_train_loss:<span class="number">.4</span>f&#125;</span>, Valid loss: <span class="subst">&#123;mean_valid_loss:<span class="number">.4</span>f&#125;</span>&#x27;</span>)</span><br><span class="line">        writer.add_scalar (<span class="string">&#x27;Loss/valid&#x27;</span>, mean_valid_loss, step)</span><br><span class="line"></span><br><span class="line">        <span class="keyword">if</span> mean_valid_loss &lt; best_loss:</span><br><span class="line">            best_loss = mean_valid_loss</span><br><span class="line">            torch.save (model.state_dict (), config [<span class="string">&#x27;save_path&#x27;</span>]) <span class="comment"># Save your best model</span></span><br><span class="line">            <span class="built_in">print</span> (<span class="string">&#x27;Saving model with loss &#123;:.3f&#125;...&#x27;</span>.<span class="built_in">format</span> (best_loss))</span><br><span class="line">            early_stop_count = <span class="number">0</span></span><br><span class="line">        <span class="keyword">else</span>: </span><br><span class="line">            early_stop_count += <span class="number">1</span></span><br><span class="line"></span><br><span class="line">        <span class="keyword">if</span> early_stop_count &gt;= config [<span class="string">&#x27;early_stop&#x27;</span>]:</span><br><span class="line">            <span class="built_in">print</span> (<span class="string">&#x27;\nModel is not improving, so we halt the training session.&#x27;</span>)</span><br><span class="line">            <span class="keyword">return</span></span><br></pre></td></tr></table></figure>
<h2 id=" 七、Configurations"><a href="# 七、Configurations" class="headerlink" title=" 七、Configurations"></a> 七、Configurations</h2><p><code>config</code> contains hyper-parameters for training and the path to save your model.</p>
<figure class="highlight python"><table><tr><td class="gutter"><pre><span class="line">1</span><br><span class="line">2</span><br><span class="line">3</span><br><span class="line">4</span><br><span class="line">5</span><br><span class="line">6</span><br><span class="line">7</span><br><span class="line">8</span><br><span class="line">9</span><br><span class="line">10</span><br><span class="line">11</span><br><span class="line">12</span><br></pre></td><td class="code"><pre><span class="line">device = <span class="string">&#x27;cuda&#x27;</span> <span class="keyword">if</span> torch.cuda.is_available () <span class="keyword">else</span> <span class="string">&#x27;cpu&#x27;</span></span><br><span class="line">config = &#123;</span><br><span class="line">    <span class="string">&#x27;seed&#x27;</span>: <span class="number">5201314</span>,      <span class="comment"># Your seed number, you can pick your lucky number. :)</span></span><br><span class="line">    <span class="string">&#x27;select_all&#x27;</span>: <span class="literal">True</span>,   <span class="comment"># Whether to use all features.</span></span><br><span class="line">    <span class="string">&#x27;valid_ratio&#x27;</span>: <span class="number">0.2</span>,   <span class="comment"># validation_size = train_size * valid_ratio</span></span><br><span class="line">    <span class="string">&#x27;n_epochs&#x27;</span>: <span class="number">3000</span>,     <span class="comment"># Number of epochs.            </span></span><br><span class="line">    <span class="string">&#x27;batch_size&#x27;</span>: <span class="number">256</span>, </span><br><span class="line">    <span class="string">&#x27;learning_rate&#x27;</span>: <span class="number">1e-5</span>,              </span><br><span class="line">    <span class="string">&#x27;early_stop&#x27;</span>: <span class="number">400</span>,    <span class="comment"># If model has not improved for this many consecutive epochs, stop training.     </span></span><br><span class="line">    <span class="string">&#x27;save_path&#x27;</span>: <span class="string">&#x27;./models/model.ckpt&#x27;</span>  <span class="comment"># Your model will be saved here.</span></span><br><span class="line">&#125;</span><br><span class="line"></span><br></pre></td></tr></table></figure>
<p>“””# Dataloader<br>Read data from files and set up training, validation, and testing sets. You do not need to modify this part.<br>“””</p>
<h1 id="Set-seed-for-reproducibility"><a href="#Set-seed-for-reproducibility" class="headerlink" title="Set seed for reproducibility"></a>Set seed for reproducibility</h1><p>same_seed (config [‘seed’])</p>
<h1 id="train-data-size-2699-x-118-id-37-states-16-features-x-5-days"><a href="#train-data-size-2699-x-118-id-37-states-16-features-x-5-days" class="headerlink" title="train_data size: 2699 x 118 (id + 37 states + 16 features x 5 days)"></a>train_data size: 2699 x 118 (id + 37 states + 16 features x 5 days)</h1><h1 id="test-data-size-1078-x-117-without-last-day’s-positive-rate"><a href="#test-data-size-1078-x-117-without-last-day’s-positive-rate" class="headerlink" title="test_data size: 1078 x 117 (without last day’s positive rate)"></a>test_data size: 1078 x 117 (without last day’s positive rate)</h1><p>train_data, test_data = pd.read_csv (‘./covid.train.csv’).values, pd.read_csv (‘./covid.test.csv’).values<br>train_data, valid_data = train_valid_split (train_data, config [‘valid_ratio’], config [‘seed’])</p>
<h1 id="Print-out-the-data-size"><a href="#Print-out-the-data-size" class="headerlink" title="Print out the data size."></a>Print out the data size.</h1><p>print (f”””train_data size: {train_data.shape}<br>valid_data size: {valid_data.shape}<br>test_data size: {test_data.shape}”””)</p>
<h1 id="Select-features"><a href="#Select-features" class="headerlink" title="Select features"></a>Select features</h1><p>x_train, x_valid, x_test, y_train, y_valid = select_feat (train_data, valid_data, test_data, config [‘select_all’])</p>
<h1 id="Print-out-the-number-of-features"><a href="#Print-out-the-number-of-features" class="headerlink" title="Print out the number of features."></a>Print out the number of features.</h1><p>print (f’number of features: {x_train.shape [1]}’)</p>
<p>train_dataset, valid_dataset, test_dataset = COVID19Dataset (x_train, y_train), \<br>                                            COVID19Dataset (x_valid, y_valid), \<br>                                            COVID19Dataset (x_test)</p>
<h1 id="Pytorch-data-loader-loads-pytorch-dataset-into-batches"><a href="#Pytorch-data-loader-loads-pytorch-dataset-into-batches" class="headerlink" title="Pytorch data loader loads pytorch dataset into batches."></a>Pytorch data loader loads pytorch dataset into batches.</h1><p>train_loader = DataLoader (train_dataset, batch_size=config [‘batch_size’], shuffle=True, pin_memory=True)<br>valid_loader = DataLoader (valid_dataset, batch_size=config [‘batch_size’], shuffle=True, pin_memory=True)<br>test_loader = DataLoader (test_dataset, batch_size=config [‘batch_size’], shuffle=False, pin_memory=True)</p>

Start training!


model = My_Model (input_dim=x_train.shape [1]).to (device) # put your model and data on the same computation device.
trainer (train_loader, valid_loader, model, config, device)

Plot learning curves with `tensorboard` (optional)

Testing

The predictions of your model on testing set will be stored at pred.csv.


def save_pred (preds, file):
    ''' Save predictions to specified file '''
    with open (file, 'w') as fp:
        writer = csv.writer (fp)
        writer.writerow (['id', 'tested_positive'])
        for i, p in enumerate (preds):
            writer.writerow ([i, p])

model = My_Model (input_dim=x_train.shape [1]).to (device)
model.load_state_dict (torch.load (config ['save_path']))
preds = predict (test_loader, model, device) 
save_pred (preds, 'pred.csv')

Reference

This notebook uses code written by Heng-Jui Chang @ NTUEE (https://github.com/ga642381/ML2021-Spring/blob/main/HW01/HW01.ipynb)