Kafka+SpringMVC+Maven应用示例

　　本文借助主流SpringMVC框架向大家介绍如何在具体应用中简单快捷的使用kafka。kafka、maven以及SpringMVC在现在的企业级应用中都占据着非常重要的地位，所以本文将三者结合起来也可以方便大家进一步熟悉基于Maven的SpringMVC框架搭建。

项目展示

　　国际惯例，首先先向大家展示一下项目最终的运行效果：

　　当项目正常启动后，在浏览器中输入：http://127.0.0.1:8080/kafkaSpringMVC/welcome 进入欢迎界面：

　　然后点击Send a Message 进入消息发送页面：

　　从上面可以看出，发送的消息是当前系统的时间（当然你也可以修改成为自己感冒的消息），点击Submit后将消息发送到kafka集群服务器，然后自动返回到Welcome欢迎界面。在欢迎界面点击Get a Message：

　　从上述界面中我们可以看见页面中已经获取到了刚才发送的消息，点击RETURN HOME，返回欢迎界面，好啦项目展示就这么简单。

开发环境

• 操作系统：MacOS 10.12.3（同样适用于Linux系统和Windows系统）
• JDK: java version "1.8.0_121"
• 开发平台：Eclipse Neon.2 Release (4.6.2)
• WEB容器：wildfly-8.1.0.Final
• zookeeper: zookeeper-3.4.9
• kafka: kafka-2.10-0.10.2.0
• maven: Eclipse Neon.2 Release(4.6.2)自带maven工具，版本为3.3.9

项目框架

　　项目框架如下图所示：

项目开发流程

　　首先搭建Maven Web Project框架，搭建过程可参考我的另一篇随笔maven web框架搭建，大家也可以在网上搜索更多的文章来学习，这里就不在重复描述啦。本项目命名为：kafkaSpringMVC。

　　框架搭建完成后，下面就需要引入Spring MVC 所需要的jar包和kafka客户端开发需要的jar包。本示例采用的Spring 版本为4.3.6.REALEASE，使用的kafka客户端版本为0.10.2.0。下面通过修改pom.xml文件来引入外部依赖包：

　　pom.xml

 <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
     <modelVersion>4.0.</modelVersion>
     <groupId>com.unionpay</groupId>
     <artifactId>kafkaSpringMVC</artifactId>
     <packaging>war</packaging>
     <version>0.0.-SNAPSHOT</version>
     <name>kafkaSpringMVC Maven Webapp</name>
     <url>http://maven.apache.org</url>

     <properties>
         <springframework>4.3..RELEASE</springframework>
     </properties>

     <dependencies>
         <dependency>
             <groupId>junit</groupId>
             <artifactId>junit</artifactId>
             <version>3.8.</version>
             <scope>test</scope>
         </dependency>

         <dependency>
             <groupId>org.springframework</groupId>
             <artifactId>spring-core</artifactId>
             <version>${springframework}</version>
         </dependency>

         <dependency>
             <groupId>org.springframework</groupId>
             <artifactId>spring-context</artifactId>
             <version>${springframework}</version>
         </dependency>

         <dependency>
             <groupId>org.springframework</groupId>
             <artifactId>spring-tx</artifactId>
             <version>${springframework}</version>
         </dependency>

         <dependency>
             <groupId>org.springframework</groupId>
             <artifactId>spring-webmvc</artifactId>
             <version>${springframework}</version>
         </dependency>

         <dependency>
             <groupId>org.springframework</groupId>
             <artifactId>spring-web</artifactId>
             <version>${springframework}</version>
         </dependency>

         <dependency>
             <groupId>org.springframework</groupId>
             <artifactId>spring-jms</artifactId>
             <version>${springframework}</version>
         </dependency>

         <dependency>
             <groupId>org.apache.kafka</groupId>
             <artifactId>kafka-clients</artifactId>
             <version>0.10.2.0</version>
         </dependency>

     </dependencies>
     <build>
         <finalName>kafkaSpringMVC</finalName>
     </build>
 </project>

　　写完pom.xml文件后，保存。然后右键项目名称，选择Maven->Update Project... 更新项目，引入jar包。

　　Update Project完成后，可以在maven依赖包里看见刚才引入的本项目需要的jar包：

　　接下来编写web.xml文件：

　　web.xml

 <?xml version="1.0" encoding="UTF-8"?>
 <web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
     xmlns="http://java.sun.com/xml/ns/javaee"
     xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
     id="WebApp_ID" version="3.0">

     <context-param>
         <param-name>contextConfigLocation</param-name>
         <param-value>classpath:kafka-beans.xml</param-value>
     </context-param>

     <listener>
         <listener-class>org.springframework.web.context.ContextLoaderListener</listener-class>
     </listener>

     <servlet>
         <servlet-name>springDispatcherServlet</servlet-name>
         <servlet-class>org.springframework.web.servlet.DispatcherServlet</servlet-class>
         <init-param>
             <param-name>contextConfigLocation</param-name>
             <param-value>classpath:spring-mvc-dispatcher.xml</param-value>
         </init-param>
         <load-on-startup></load-on-startup>
     </servlet>

     <servlet-mapping>
         <servlet-name>springDispatcherServlet</servlet-name>
         <url-pattern>/</url-pattern>
     </servlet-mapping>

 </web-app>

　　简单明了，从web.xml描述来看，MVC映射是通过spring-mvc-dispatcher.xml文件说明的，而项目中用到的所有的model 则是通过kafka-beans.xml文件注入的。那么下面分别在resources文件夹下建立这两个xml配置文件：

　　spring-mvc-dispatcher.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context"
    xmlns:mvc="http://www.springframework.org/schema/mvc"
    xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-4.0.xsd
        http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc-4.0.xsd">

    <context:component-scan base-package="com.unionpay.controller"></context:component-scan>

    <bean
        class="org.springframework.web.servlet.view.InternalResourceViewResolver">
        <property name="prefix" value="/WEB-INF/views/"></property>
        <property name="suffix" value=".jsp"></property>
    </bean>

</beans>

　　就这么简单。由于kafka-beans.xml主要作用是配置注入的beans，所以按照程序，还是先建立好Model才符合逻辑。

　　在src/main/java路径下分别建立三个package包：com.unionpay.producer、com.unionpay.consumer、com.unionpay.controller。从字面意思很容易理解，producer包主要用于存放生产者，consumer包主要用于存放消费者，controller包主要用于存放逻辑控制类。

　　完成后在com.unionpay.producer包下建立KafkaProducerDemo.java文件（最好不要命名为Producer或者KafkaProducer，这样会与引入的jar包中原有的类重名）：

　　KafkaProducerDemo.java

 package com.unionpay.producer;

 import java.util.Properties;

 import org.apache.kafka.clients.producer.KafkaProducer;
 import org.apache.kafka.clients.producer.ProducerRecord;

 public class KafkaProducerDemo {

     Properties properties;

     public KafkaProducerDemo(Properties properties) {
         super();
         this.properties = properties;
     }

     public Properties getProperties() {
         return properties;
     }

     public void setProperties(Properties properties) {
         this.properties = properties;
     }

     public void sendMessage(String msg) {

         KafkaProducer<String, String> producer = new KafkaProducer<String, String>(properties);

         ProducerRecord<String, String> record = new ProducerRecord<String, String>(properties.getProperty("topic"),
                 msg);
         producer.send(record);

         producer.close();

     }

 }

　　Properties属性主要是为了配置KafkaProducer类，具体信息通过Spring注入，这样可以显得更加高大上和灵活。后面的配置文件中我们可以看得到具体的配置信息。

　　同样，在com.unionpay.consumer包下面新建类KafkaConsumerDemo.java:

 package com.unionpay.consumer;

 import java.util.Arrays;
 import java.util.Properties;

 import org.apache.kafka.clients.consumer.ConsumerRecord;
 import org.apache.kafka.clients.consumer.ConsumerRecords;
 import org.apache.kafka.clients.consumer.KafkaConsumer;

 public class KafkaConsumerDemo {

     private Properties props;

     public KafkaConsumerDemo(Properties props) {
         super();
         this.props = props;
     }

     public Properties getProps() {
         return props;
     }

     public void setProps(Properties props) {
         this.props = props;
     }

     public String receive(){

         KafkaConsumer<String,String> consumer = new KafkaConsumer<String,String>(props);
         consumer.subscribe(Arrays.asList(props.getProperty("topic")));

         String msg = "";
         while(true){
             ConsumerRecords<String,String> consumerRecords = consumer.poll();
             for(ConsumerRecord<String, String> consumerRecord:consumerRecords){
                 msg += consumerRecord.value();
             }
             consumer.close();
             return msg;
         }
     }

 }

　　也是基于同样的原因，KafkaConsumer的配置信息properties也是通过Spring配置文件注入。

　　当Producer和Consumer编写完成后，就可以编写kafka-beans.xml文件啦：

　　kafka-beans.xml

 <?xml version="1.0" encoding="UTF-8"?>
 <beans xmlns="http://www.springframework.org/schema/beans"
     xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:context="http://www.springframework.org/schema/context"
     xmlns:mvc="http://www.springframework.org/schema/mvc"
     xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd
         http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-4.0.xsd
         http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc-4.0.xsd">

     <context:component-scan base-package="com.unionpay.producer"></context:component-scan>
     <context:component-scan base-package="com.unionpay.consumer"></context:component-scan>

     <bean id="kafkaProducerDemo" class="com.unionpay.producer.KafkaProducerDemo">
         <property name="properties">
             <props>
                 <prop key="topic">my-replicated-topic</prop>
                 <prop key="bootstrap.servers">127.0.0.1:</prop>
                 <prop key="acks">all</prop>
                 <prop key="key.serializer">org.apache.kafka.common.serialization.StringSerializer
                 </prop>
                 <prop key="value.serializer">org.apache.kafka.common.serialization.StringSerializer
                 </prop>
                 <prop key="buffer.memory"></prop>
             </props>

         </property>
     </bean>

     <bean id="kafkaConsumerDemo" class="com.unionpay.consumer.KafkaConsumerDemo">
         <property name="props">
             <props>
                 <prop key="topic">my-replicated-topic</prop>
                 <prop key="bootstrap.servers">127.0.0.1:</prop>
                 <prop key="group.id">group1</prop>
                 <prop key="enable.auto.commit">true</prop>
                 <prop key="auto.commit.interval.ms"></prop>
                 <prop key="session.timeout.ms"></prop>
                 <prop key="key.deserializer">org.apache.kafka.common.serialization.StringDeserializer
                 </prop>
                 <prop key="value.deserializer">org.apache.kafka.common.serialization.StringDeserializer
                 </prop>
             </props>

         </property>
     </bean>
 </beans>

　　kafka的主要配置一共有三种：broker、producer和consumer，对于客户端来说就是后两种啦。而后两种的配置项从官方文件可以知道，每个都至少有30多种配置内容。通过上面这种注入配置方式的话，在<props><props>中随便添加配置内容，是不是很灵活呢^_^

　　下面在com.unionpay.controller包下编写Controller类，控制业务逻辑：

　　KafkaController.java

package com.unionpay.controller;

import java.text.SimpleDateFormat;
import java.util.Date;

import javax.annotation.Resource;

import org.springframework.stereotype.Controller;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.servlet.ModelAndView;

import com.unionpay.consumer.KafkaConsumerDemo;
import com.unionpay.producer.KafkaProducerDemo;

@Controller
public class KafkaController {

    @Resource(name = "kafkaProducerDemo")
    KafkaProducerDemo producer;

    @Resource(name = "kafkaConsumerDemo")
    KafkaConsumerDemo consumer;

    @RequestMapping(value = "/welcome")
    public ModelAndView welcome() {
        System.out.println("--------welcome--------");
        ModelAndView mv = new ModelAndView();
        mv.setViewName("welcome");
        return mv;
    }

    @RequestMapping(value = "/sendmessage", method = RequestMethod.GET)
    public ModelAndView sendMessage() {
        System.out.println("--------sendmessage--------");
        Date date = new Date();
        SimpleDateFormat sdf = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        String now = sdf.format(date);

        ModelAndView mv = new ModelAndView();
        mv.addObject("time", now);
        mv.setViewName("kafka_send");
        return mv;
    }

    @RequestMapping(value = "/onsend", method = RequestMethod.POST)
    public ModelAndView onsend(@RequestParam("message") String msg) {
        System.out.println("--------onsend--------");
        producer.sendMessage(msg);

        ModelAndView mv = new ModelAndView();
        mv.setViewName("welcome");
        return mv;
    }

    @RequestMapping(value = "/receive")
    public ModelAndView receive() {
        System.out.println("--------receive--------");

        String msg = consumer.receive();

        ModelAndView mv = new ModelAndView();
        mv.addObject("msg", msg);
        mv.setViewName("kafka_receive");
        return mv;
    }

}

　　到目前为止，我们的MVC中已经完成了两个啦（M and C），下面编写最后的三个JSP文件。从解析配置文件（spring-mvc-dispatcher.xml）来看，我们的JSP页面应该建立在/WEB-INF/views/目录下，所以我们首先在/WEB-INF/目录下建立views文件夹。然后在该文件夹下面建立三个jsp文件：

　　welcome.jsp

 <%@ page language="java" contentType="text/html; charset=UTF-8"
     pageEncoding="UTF-8"%>
 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
 <html>
 <head>
 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
 <title>welcome</title>
 </head>
 <body>
     <h1>Welcome</h1>
     <h2><a href="sendmessage">Send a Message</a></h2>
     <h2><a href="receive">Get a Message</a></h2>
 </body>
 </html>

　　kafka_send.jsp

 <%@ page language="java" contentType="text/html; charset=UTF-8"
     pageEncoding="UTF-8"%>
 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
 <html>
 <head>
 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
 <title>kafka_send</title>
 </head>
 <body>
     <h1>Send a Message</h1>
     <form action="onsend" method="post">
     MessageText:<textarea name="message">${time}</textarea>
     <br>
     <input type="submit" value="Submit">
     </form>

     <h2><a href="welcome">RETURN HOME</a></h2>

 </body>
 </html>

　　kafka-receive.jsp

 <%@ page language="java" contentType="text/html; charset=UTF-8"
     pageEncoding="UTF-8"%>
 <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
 <html>
 <head>
 <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
 <title>kafka_receive</title>
 </head>
 <body>

     <h1>Kafka_Reveive!!!</h1>
     <h2>Receive Message : ${msg}</h2>
     <h2><a href="welcome">RETURN HOME</a></h2>
 </body>
 </html>

　　啊，终于大功告成啦。下面就要品尝我们的劳动果实啦。将项目部署在容器里，然后首先启动zookeeper集群服务器，然后启动kafka集群服务器：

//启动zookeeper集群服务器
cd ~/DevelopEnvironment/zookeeper-3.4.-kafka/bin
./zkServer.sh start

//启动kafka集群服务器
cd ~/DevelopEnvironment/kafka_2.-0.10.2.0/bin
./kafka-server-start.sh ../config/server.properties
./kafka-server-start.sh ../config/server-.properties
./kafka-server-start.sh ../config/server-.properties

　　然后通过Eclipse启动容器：

　　

　　从上面终端打印出来的信息可以知道，部署成功啦。下面就要轮回到本文开头啦，在网页地址栏中输入：http://127.0.0.1:8080/kafkaSpringMVC/welcome 进入欢迎界面，然后按照开始描述的操作进行操作，看看能否成功发送和接受消息呢？

　　怎么样，你成功了吗？反正我是成功了，也希望你也成功啦。如果出现什么错误的话也千万别着急，去根据报错信息找找原因，因为你也马上就要成功啦。

　　源码下载：kafkaSpringMVC.zip

参考文献

• kafkaProducer官方参考文档
• kafkaConsumer官方参考文档