hadoop运行原理之Job运行(三) TaskTracker的启动及初始化

　　与JobTracker一样，TaskTracker也有main()方法，然后以线程的方式启动(继承了Runnable接口)。main()方法中主要包含两步：一是创建一个TaskTracker对象；二是启动TaskTracker线程。

 public static void main(String argv[]) throws Exception {
     ...
     try {
       JobConf conf=new JobConf();
       ...
       TaskTracker tt = new TaskTracker(conf);
       ...
       tt.run();
     } catch (Throwable e) {
      ...
     }
   }

　　一、TaskTracker tt = new TaskTracker(conf);  该方法会首先获取Map和Reduce的slot数量、磁盘检查时间间隔和JobTracker的地址；然后构造一个HttpServer(Jetty服务器)对象，这样可以通过web查看该tasktracker的运行情况及所有task；接着构造TaskController对象，用来控制任务的初始化、终结和清理工作；再接着调用initialize()方法来初始化一些变量并构造TaskTracker对象；最后启动HttpServe服务器。

　　着重说下initialize()方法。这个方法真正完成了构造TaskTracker对象的工作，并初始化了一些变量。用这个单独的方法来构造TaskTracker，可以使我们多次调用它。首先initialize()方法会清理一些历史文件(第一次启动TaskTracker不会有历史文件)，并新建一些目录；接着初始化一些变量；构造JvmManager对象用来管理JVM；初始化并启动taskReportServer，Task通过RPC向该Server汇报进度；初始化distributedCacheManager对象用来管理分布式缓存；初始化InterTrackerProtocol对象，TaskTracker由此与JobTracker通信；构造两个TaskLauncher线程：mapLauncher和reduceLauncher并启动，这个线程的run函数就是不断监测taskToLaunch队列中是否有新的 TaskTracker.TaskInProgress对象加入。如果有则从中取出一个对象，然后调用TaskTracker类的 startNewTask(TaskInProgress tip)来启动一个task。

　　二、tt.run();  该方法维护一个与JobTracker的通信连接，以周期性地向JobTracker发送心跳并领取新任务。

 public void run() {
     try {
       getUserLogManager().start();
       startCleanupThreads();
       boolean denied = false;
       while (running && !shuttingDown) {
         boolean staleState = false;
         try {
           // This while-loop attempts reconnects if we get network errors
           while (running && !staleState && !shuttingDown && !denied) {    //该循环条件表示与JobTracker连接成功；
               //否则的话会在外层循环中一直尝试执行该内层循环
             try {
               //如果连接JobTracker服务成功，TaskTracker就会调用offerService()函数进入主执行循环中。
               //offerService()中的循环会每隔一段时间与JobTracker通讯一次，调用transmitHeartBeat()，获得HeartbeatResponse信息。
               State osState = offerService();
               if (osState == State.STALE) {
                 staleState = true;
               } else if (osState == State.DENIED) {
                 denied = true;
               }
             } catch (Exception ex) {
               if (!shuttingDown) {
                 LOG.info("Lost connection to JobTracker [" +
                          jobTrackAddr + "].  Retrying...", ex);
                 try {
                   Thread.sleep(5000);
                 } catch (InterruptedException ie) {
                 }
               }
             }
           }
         } finally {
           // If denied we'll close via shutdown below. We should close
           // here even if shuttingDown as shuttingDown can be set even
           // if shutdown is not called.
           if (!denied) {
             close();
           }
         }
         if (shuttingDown) { return; }
         if (denied) { break; }
         LOG.warn("Reinitializing local state");
         initialize();
       }
       if (denied) {
         shutdown();    //关闭该TaskTracker
       }
     } catch (IOException iex) {
       LOG.error("Got fatal exception while reinitializing TaskTracker: " +
                 StringUtils.stringifyException(iex));
       return;
     } catch (InterruptedException i) {
       LOG.error("Got interrupted while reinitializing TaskTracker: " +
           i.getMessage());
       return;
     }
   }

　　该方法主体是两层while循环。如果与JobTracker的连接一旦断开，则反复尝试运行内层循环，直到与JobTracker成功连接为止；连接成功后会调用内层循环中的offerService()方法与JobTracker进行通信。

 State offerService() throws Exception {
     long lastHeartbeat = System.currentTimeMillis();    //上次心跳时间

     while (running && !shuttingDown) {
       try {
         long now = System.currentTimeMillis();

         // accelerate to account for multiple finished tasks up-front
         //计算上次心跳到现在的时间隔
         //通过完成的任务数动态调整时间间隔
         long remaining =
           (lastHeartbeat + getHeartbeatInterval(finishedCount.get())) - now;

         if (remaining <= 0) {    //达到心跳时间间隔
           finishedCount.set(0);
         }

         while (remaining > 0) {    //未达到心跳时间间隔
           // sleeps for the wait time or
           // until there are *enough* empty slots to schedule tasks
           synchronized (finishedCount) {
             finishedCount.wait(remaining);    //wait相差的时间(remaining)

             // Recompute
             now = System.currentTimeMillis();
             remaining =
               (lastHeartbeat + getHeartbeatInterval(finishedCount.get())) - now;

             if (remaining <= 0) {
               // Reset count
               finishedCount.set(0);
               break;
             }
           }
         }

         // If the TaskTracker is just starting up:
         // 1. Verify the versions matches with the JobTracker
         // 2. Get the system directory & filesystem
         if(justInited) {
           String jtBuildVersion = jobClient.getBuildVersion();
           String jtVersion = jobClient.getVIVersion();
           if (!isPermittedVersion(jtBuildVersion, jtVersion)) {
             String msg = "Shutting down. Incompatible version or build version." +
               "TaskTracker version '" + VersionInfo.getVersion() +
               "' and build '" + VersionInfo.getBuildVersion() +
               "' and JobTracker version '" + jtVersion +
               "' and build '" + jtBuildVersion +
               " and " + CommonConfigurationKeys.HADOOP_RELAXED_VERSION_CHECK_KEY +
               " is " + (relaxedVersionCheck ? "enabled" : "not enabled") +
               " and " + CommonConfigurationKeys.HADOOP_SKIP_VERSION_CHECK_KEY +
               " is " + (skipVersionCheck ? "enabled" : "not enabled");
             LOG.fatal(msg);
             try {
               jobClient.reportTaskTrackerError(taskTrackerName, null, msg);
             } catch(Exception e ) {
               LOG.info("Problem reporting to jobtracker: " + e);
             }
             return State.DENIED;
           }

           String dir = jobClient.getSystemDir();
           while (dir == null) {
             LOG.info("Failed to get system directory...");

             // Re-try
             try {
               // Sleep interval: 1000 ms - 5000 ms
               int sleepInterval = 1000 + r.nextInt(4000);
               Thread.sleep(sleepInterval);
             } catch (InterruptedException ie)
             {}
             dir = jobClient.getSystemDir();
           }
           systemDirectory = new Path(dir);
           systemFS = systemDirectory.getFileSystem(fConf);
         }

         now = System.currentTimeMillis();
         if (now > (lastCheckDirsTime + diskHealthCheckInterval)) {    //是否达到磁盘检查间隔时间
           localStorage.checkDirs();
           lastCheckDirsTime = now;
           int numFailures = localStorage.numFailures();
           // Re-init the task tracker if there were any new failures
           if (numFailures > lastNumFailures) {
             lastNumFailures = numFailures;
             return State.STALE;
           }
         }

         // Send the heartbeat and process the jobtracker's directives
         //发送Heartbeat到JobTracker，得到response
         HeartbeatResponse heartbeatResponse = transmitHeartBeat(now);

         // Note the time when the heartbeat returned, use this to decide when to send the
         // next heartbeat
         lastHeartbeat = System.currentTimeMillis();

         // Check if the map-event list needs purging
         Set<JobID> jobs = heartbeatResponse.getRecoveredJobs();
         if (jobs.size() > 0) {
           synchronized (this) {
             // purge the local map events list
             for (JobID job : jobs) {
               RunningJob rjob;
               synchronized (runningJobs) {
                 rjob = runningJobs.get(job);
                 if (rjob != null) {
                   synchronized (rjob) {
                     FetchStatus f = rjob.getFetchStatus();
                     if (f != null) {
                       f.reset();
                     }
                   }
                 }
               }
             }

             // Mark the reducers in shuffle for rollback
             synchronized (shouldReset) {
               for (Map.Entry<TaskAttemptID, TaskInProgress> entry
                    : runningTasks.entrySet()) {
                 if (entry.getValue().getStatus().getPhase() == Phase.SHUFFLE) {
                   this.shouldReset.add(entry.getKey());
                 }
               }
             }
           }
         }

         //调用HeartbeatResponse的getActions()函数获得JobTracker传过来的所有指令，即一个TaskTrackerAction数组
         TaskTrackerAction[] actions = heartbeatResponse.getActions();
         if(LOG.isDebugEnabled()) {
           LOG.debug("Got heartbeatResponse from JobTracker with responseId: " +
                     heartbeatResponse.getResponseId() + " and " +
                     ((actions != null) ? actions.length : 0) + " actions");
         }
         if (reinitTaskTracker(actions)) {
           return State.STALE;
         }

         // resetting heartbeat interval from the response.
         heartbeatInterval = heartbeatResponse.getHeartbeatInterval();
         justStarted = false;
         justInited = false;
         if (actions != null){
         //遍历这个数组
           for(TaskTrackerAction action: actions) {
             if (action instanceof LaunchTaskAction) {    //如果是LaunchTaskAction，则调用调用addToTaskQueue加入到待执行队列
               addToTaskQueue((LaunchTaskAction)action);
             } else if (action instanceof CommitTaskAction) {    //如果是CommitTaskAction，则添加到需要提交的队列中
               CommitTaskAction commitAction = (CommitTaskAction)action;
               if (!commitResponses.contains(commitAction.getTaskID())) {
                 LOG.info("Received commit task action for " +
                           commitAction.getTaskID());
                 commitResponses.add(commitAction.getTaskID());
               }
             } else {    //如果是KillJobAction或者KillTaskAction，则加入到tasksToCleanup队列，交给一个taskCleanupThread线程来处理
               addActionToCleanup(action);
             }
           }
         }
         //杀死一定时间没没有汇报进度的task
         markUnresponsiveTasks();

         //当剩余磁盘空间小于mapred.local.dir.minspacekill(默认为0)时，寻找合适的任务将其杀掉以释放空间
         killOverflowingTasks();

         //we've cleaned up, resume normal operation
         if (!acceptNewTasks && isIdle()) {
           acceptNewTasks=true;
         }
         //The check below may not be required every iteration but we are
         //erring on the side of caution here. We have seen many cases where
         //the call to jetty's getLocalPort() returns different values at
         //different times. Being a real paranoid here.
         checkJettyPort(server.getPort());
       } catch (InterruptedException ie) {
         LOG.info("Interrupted. Closing down.");
         return State.INTERRUPTED;
       } catch (DiskErrorException de) {
         String msg = "Exiting task tracker for disk error:\n" +
           StringUtils.stringifyException(de);
         LOG.error(msg);
         synchronized (this) {
           jobClient.reportTaskTrackerError(taskTrackerName,
                                            "DiskErrorException", msg);
         }
         // If we caught a DEE here we have no good dirs, therefore shutdown.
         return State.DENIED;
       } catch (RemoteException re) {
         String reClass = re.getClassName();
         if (DisallowedTaskTrackerException.class.getName().equals(reClass)) {
           LOG.info("Tasktracker disallowed by JobTracker.");
           return State.DENIED;
         }
       } catch (Exception except) {
         String msg = "Caught exception: " +
           StringUtils.stringifyException(except);
         LOG.error(msg);
       }
     }

     return State.NORMAL;
   }

　　通过下面这个图可以很清楚的了解这个方法的功能：

　　该方法的主体是一个while循环，首先检查是否达到心跳间隔时间，如果没有的话则阻塞一定时间(wait)；然后检查TaskTracker是否刚启动，如果是的话需要先检查JobTracker版本和TaskTracker版本是否一致，不一致的话返回State.DENIED；再获取system directory和filesystem；接着如果达到磁盘检查时间，则检查磁盘是否有损坏；再接着transmitHeartBeat(now)方法将当前结点的运行时信息(如资源使用情况、任务运行状态等)，通过心跳汇报给JobTracker，同时接收来自JobTracker的各种命令；JobTracker将心跳应答封装在HeartbeatResponse对象中，该对象包括两部分内容：第一是作业集合recoveredJobs，它是上次关闭JobTracker时正在运行的作业集合，重启JobTracker后需要恢复这些作业的运行状态；第二是需要执行的命令列表actions。接着会检查这些命令里是否包括了重新初始化命令(ReinitTrackerAction)，如果有则返回State.STALE，会重新初始化TaskTracker；修正心跳间隔；遍历actions数组，如果是LaunchTaskAction，则根据是MapTask还是ReduceTask，分别放进相应的mapLauncher和reduceLauncher中，这两个launcher都是TaskLauncher；如果是CommitTaskAction则加入到commitResponses；其他的类型放入tasksToCleanup.put(action)，表示要清理，如执行KillJobAction或者KillTaskAction等。kUnresponsiveTasks()方法杀死一定时间内没有汇报进度的任务；killOverflowingTasks()中，当剩余磁盘空间小于mapred.local.dir.minspacekill(默认为0)时，寻找合适的任务将其杀掉以释放空间(优先杀掉Reduce任务)。

　　下面分析下transmitHeartBeat(now)方法，这是offerService()方法中最重要的步骤。还是先来看个图，这样能对这个方法的流程有个大概了解。

　　transmitHeartBeat(now)方法主要是构造心跳信息(会把当前结点信息和该结点上的所有task信息告知JobTracker)并向JobTracker发送心跳的。首先它会检查是否达到计数器时间间隔(60s)，如果没有达到的话，那么就不发送任务的计数器(Counter)信息，否则要发送。然后检查上次心跳是否结束，如果没有结束，则继续使用上次的结点状态信息；如果结束的话则构造新的结点状态信息(会把当前结点的健康状态、资源情况以及task信息封装到里边)。接着判断该TaskTracker是否能够接收新任务，如果能够接收新任务，则将当前结点的起源情况设置到status对象里。接着再设置结点的健康状态。结点的健康状态由NodeHealthCheckerService线程来监测，该线程运行一个脚本来监控结点的运行情况；如果该线程监测到脚本输出里有“ERROR”开头行，则将结点标记为“unhealthy”；如果没有设置该脚本时，则不启动该线程，结点默认“healthy”；最后通过RPC调用JobTracker的heartbeat()方法。在得到心跳响应后，将status标记为null，表示本次心跳通信结束，并返回心跳响应。

　　关于TaskTracker的一个思考。TaskTracker与JobTracker不同的是，TaskTracker是以线程的方式启动的，而JobTracker却不是。这是否也说明了Hadoop采用的pull的模式，从这儿看出TaskTracker主动与JobTracker进行通信？

　　本文基于hadoop1.2.1

　　如有错误，还请指正

　　参考文章：《Hadoop技术内幕 深入理解MapReduce架构设计与实现原理》 董西成

　　转载请注明出处：http://www.cnblogs.com/gwgyk/p/3999811.html