Solr4.8.0源码分析(7)之Solr SPI

Solr4.8.0源码分析(7)之Solr SPI

查看Solr源码时候会发现，每一个package都会由对应的resources. 如下图所示：

一时对这玩意好奇了，看了文档以后才发现，这个services就是java SPI机制。首先介绍下java SPI机制，然后再结合Solr谈一下SPI。

1. JAVA SPI

当服务的提供者，提供了服务接口的一种实现之后，在jar包的META-INF/services/目录里同时创建一个以服务接口命名的文件。该文件里就是实现该服务接口的具体实现类。而当外部程序装配这个模块的时候，就能通过该jar包META-INF/services/里的配置文件找到具体的实现类名，并装载实例化，完成模块的注入。

基于这样一个约定就能很好的找到服务接口的实现类，而不需要再代码里制定。

jdk提供服务实现查找的一个工具类：java.util.ServiceLoader

假设有一个内容搜索系统，分为展示和搜索两个模块。展示和搜索基于接口编程。搜索的实现可能是基于文件系统的搜索，也可能是基于数据库的搜索。实例代码如下：

Search.java: 搜索接口

 package search;

 import java.util.List;

 import definition.Doc;

 public interface Search {
     List<Doc> search(String keyword);
 }

FileSearch.java:文件系统的搜索实现

 package search;

 import java.util.List;

 import definition.Doc;

 public class FileSearch implements Search {

     @Override
     public List<Doc> search(String keyword) {
         System.out.println("now use file system search. keyword:" + keyword);
         return null;
     }

 }

DatabaseSearch.java

 package search;

 import java.util.List;

 import definition.Doc;

 public class DatabaseSearch implements Search {

     @Override
     public List<Doc> search(String keyword) {
         System.out.println("now use database search. keyword:" + keyword);
         return null;
     }

 }

SearchTest.java

 package search;

 import java.util.Iterator;
 import java.util.ServiceLoader;

 public class SearchTest {

     public static void main(String[] args) {
         ServiceLoader<Search> s = ServiceLoader.load(Search.class);
         Iterator<Search> searchs = s.iterator();
         if (searchs.hasNext()) {
             Search curSearch = searchs.next();
             curSearch.search("test");
         }
     }
 }

最后创建在META-INF/searvices/search.Search文件。

当search.Search文件内容是"search.FileSearch"时，程序输出是：

now use file system search. keyword:test

当search.Search文件内容是"search.DatabaseSearch"时，程序输出是：

now use database search. keyword:test 
可以看出SearchTest里没有任何和具体实现有关的代码，而是基于spi的机制去查找服务的实现。

2. Solr SPI

以Codec类为例，查看resources/META-INF/services/org.apache.lucene.codecs.Codec:可以看出Codec服务接口具有以下具体的实现类。这就很好的解释了Solrconfig.xml里面的LuceneVersion的配置，也为Lucene的向前兼容提供了保障。

 #  Licensed to the Apache Software Foundation (ASF) under one or more
 #  contributor license agreements.  See the NOTICE file distributed with
 #  this work for additional information regarding copyright ownership.
 #  The ASF licenses this file to You under the Apache License, Version 2.0
 #  (the "License"); you may not use this file except in compliance with
 #  the License.  You may obtain a copy of the License at
 #
 #       http://www.apache.org/licenses/LICENSE-2.0
 #
 #  Unless required by applicable law or agreed to in writing, software
 #  distributed under the License is distributed on an "AS IS" BASIS,
 #  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 #  See the License for the specific language governing permissions and
 #  limitations under the License.

 org.apache.lucene.codecs.lucene40.Lucene40Codec
 org.apache.lucene.codecs.lucene3x.Lucene3xCodec
 org.apache.lucene.codecs.lucene41.Lucene41Codec
 org.apache.lucene.codecs.lucene42.Lucene42Codec
 org.apache.lucene.codecs.lucene45.Lucene45Codec
 org.apache.lucene.codecs.lucene46.Lucene46Codec

接下来可以看下Codec服务接口的实现代码

 package org.apache.lucene.codecs;

 /*
  * Licensed to the Apache Software Foundation (ASF) under one or more
  * contributor license agreements.  See the NOTICE file distributed with
  * this work for additional information regarding copyright ownership.
  * The ASF licenses this file to You under the Apache License, Version 2.0
  * (the "License"); you may not use this file except in compliance with
  * the License.  You may obtain a copy of the License at
  *
  *     http://www.apache.org/licenses/LICENSE-2.0
  *
  * Unless required by applicable law or agreed to in writing, software
  * distributed under the License is distributed on an "AS IS" BASIS,
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */

 import java.util.Set;
 import java.util.ServiceLoader; // javadocs

 import org.apache.lucene.index.IndexWriterConfig; // javadocs
 import org.apache.lucene.util.NamedSPILoader;

 /**
  * Encodes/decodes an inverted index segment.
  * <p>
  * Note, when extending this class, the name ({@link #getName}) is
  * written into the index. In order for the segment to be read, the
  * name must resolve to your implementation via {@link #forName(String)}.
  * This method uses Java's
  * {@link ServiceLoader Service Provider Interface} (SPI) to resolve codec names.
  * <p>
  * If you implement your own codec, make sure that it has a no-arg constructor
  * so SPI can load it.
  * @see ServiceLoader
  */
 public abstract class Codec implements NamedSPILoader.NamedSPI {

   private static final NamedSPILoader<Codec> loader =
     new NamedSPILoader<>(Codec.class);

   private final String name;

   /**
    * Creates a new codec.
    * <p>
    * The provided name will be written into the index segment: in order to
    * for the segment to be read this class should be registered with Java's
    * SPI mechanism (registered in META-INF/ of your jar file, etc).
    * @param name must be all ascii alphanumeric, and less than 128 characters in length.
    */
   protected Codec(String name) {
     NamedSPILoader.checkServiceName(name);
     this.name = name;
   }

   /** Returns this codec's name */
   @Override
   public final String getName() {
     return name;
   }
   /**
    * 以下几个Format跟Lucene的索引文件格式有关
    * */
   /** Encodes/decodes postings */
   public abstract PostingsFormat postingsFormat();

   /** Encodes/decodes docvalues */
   public abstract DocValuesFormat docValuesFormat();

   /** Encodes/decodes stored fields */
   public abstract StoredFieldsFormat storedFieldsFormat();

   /** Encodes/decodes term vectors */
   public abstract TermVectorsFormat termVectorsFormat();

   /** Encodes/decodes field infos file */
   public abstract FieldInfosFormat fieldInfosFormat();

   /** Encodes/decodes segment info file */
   public abstract SegmentInfoFormat segmentInfoFormat();

   /** Encodes/decodes document normalization values */
   public abstract NormsFormat normsFormat();

   /** Encodes/decodes live docs */
   public abstract LiveDocsFormat liveDocsFormat();

   /**
    * 根据名字在已有的Codec实例中寻找符合
    * */
   /** looks up a codec by name */
   public static Codec forName(String name) {
     if (loader == null) {
       throw new IllegalStateException("You called Codec.forName() before all Codecs could be initialized. "+
           "This likely happens if you call it from a Codec's ctor.");
     }
     return loader.lookup(name);
   }

   /**
    * 返回有效的Codecs实例
    * */
   /** returns a list of all available codec names */
   public static Set<String> availableCodecs() {
     if (loader == null) {
       throw new IllegalStateException("You called Codec.availableCodecs() before all Codecs could be initialized. "+
           "This likely happens if you call it from a Codec's ctor.");
     }
     return loader.availableServices();
   }

   /**
    * 更新Codec实例列表，Codec实例列表只能添加，不能删除与更改。
    * */
   /**
    * Reloads the codec list from the given {@link ClassLoader}.
    * Changes to the codecs are visible after the method ends, all
    * iterators ({@link #availableCodecs()},...) stay consistent.
    *
    * <p><b>NOTE:</b> Only new codecs are added, existing ones are
    * never removed or replaced.
    *
    * <p><em>This method is expensive and should only be called for discovery
    * of new codecs on the given classpath/classloader!</em>
    */
   public static void reloadCodecs(ClassLoader classloader) {
     loader.reload(classloader);
   }

   /**
    * 默认为Lucene46，也就是说默认调用的是org.apache.lucene.codecs.lucene46.Lucene46Codec
    * */
   private static Codec defaultCodec = Codec.forName("Lucene46");

   /**
    * 返回默认的Codec实例
    * */
   /** expert: returns the default codec used for newly created
    *  {@link IndexWriterConfig}s.
    */
   // TODO: should we use this, or maybe a system property is better?
   public static Codec getDefault() {
     return defaultCodec;
   }

   /**
    * 设置默认的Codec实例
    * */
   /** expert: sets the default codec used for newly created
    *  {@link IndexWriterConfig}s.
    */
   public static void setDefault(Codec codec) {
     defaultCodec = codec;
   }

   /**
    * returns the codec's name. Subclasses can override to provide
    * more detail (such as parameters).
    */
   @Override
   public String toString() {
     return name;
   }
 }

代码比较简单明了，接下来再看下NamedSPILoader.NamedSPI，它封装了JAVA SPI的实现：

 package org.apache.lucene.util;

 /*
  * Licensed to the Apache Software Foundation (ASF) under one or more
  * contributor license agreements.  See the NOTICE file distributed with
  * this work for additional information regarding copyright ownership.
  * The ASF licenses this file to You under the Apache License, Version 2.0
  * (the "License"); you may not use this file except in compliance with
  * the License.  You may obtain a copy of the License at
  *
  *     http://www.apache.org/licenses/LICENSE-2.0
  *
  * Unless required by applicable law or agreed to in writing, software
  * distributed under the License is distributed on an "AS IS" BASIS,
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */

 import java.util.Collections;
 import java.util.Iterator;
 import java.util.Map;
 import java.util.LinkedHashMap;
 import java.util.Set;
 import java.util.ServiceConfigurationError;

 /**
  * Helper class for loading named SPIs from classpath (e.g. Codec, PostingsFormat).
  * @lucene.internal
  */
 public final class NamedSPILoader<S extends NamedSPILoader.NamedSPI> implements Iterable<S> {

   /**
    * SPI service Map,存放服务对应的实例类。
    * */
   private volatile Map<String,S> services = Collections.emptyMap();
   private final Class<S> clazz;

   public NamedSPILoader(Class<S> clazz) {
     this(clazz, Thread.currentThread().getContextClassLoader());
   }

   public NamedSPILoader(Class<S> clazz, ClassLoader classloader) {
     this.clazz = clazz;
     // if clazz' classloader is not a parent of the given one, we scan clazz's classloader, too:
     final ClassLoader clazzClassloader = clazz.getClassLoader();
     if (clazzClassloader != null && !SPIClassIterator.isParentClassLoader(clazzClassloader, classloader)) {
       reload(clazzClassloader);
     }
     reload(classloader);
   }

   /**
    * 更新SPI MAP services。遍历META-INF/services文件，如果services MAP没有该实例，则新建实例，并放入services MAP
    * */
   /**
    * Reloads the internal SPI list from the given {@link ClassLoader}.
    * Changes to the service list are visible after the method ends, all
    * iterators ({@link #iterator()},...) stay consistent.
    *
    * <p><b>NOTE:</b> Only new service providers are added, existing ones are
    * never removed or replaced.
    *
    * <p><em>This method is expensive and should only be called for discovery
    * of new service providers on the given classpath/classloader!</em>
    */
   public synchronized void reload(ClassLoader classloader) {
     final LinkedHashMap<String,S> services = new LinkedHashMap<>(this.services);
     final SPIClassIterator<S> loader = SPIClassIterator.get(clazz, classloader);
     while (loader.hasNext()) {
       final Class<? extends S> c = loader.next();
       try {
         final S service = c.newInstance();
         final String name = service.getName();
         // only add the first one for each name, later services will be ignored
         // this allows to place services before others in classpath to make
         // them used instead of others
         if (!services.containsKey(name)) {
           checkServiceName(name);
           services.put(name, service);
         }
       } catch (Exception e) {
         throw new ServiceConfigurationError("Cannot instantiate SPI class: " + c.getName(), e);
       }
     }
     this.services = Collections.unmodifiableMap(services);
   }

   /**
    * Validates that a service name meets the requirements of {@link NamedSPI}
    */
   public static void checkServiceName(String name) {
     // based on harmony charset.java
     if (name.length() >= 128) {
       throw new IllegalArgumentException("Illegal service name: '" + name + "' is too long (must be < 128 chars).");
     }
     for (int i = 0, len = name.length(); i < len; i++) {
       char c = name.charAt(i);
       if (!isLetterOrDigit(c)) {
         throw new IllegalArgumentException("Illegal service name: '" + name + "' must be simple ascii alphanumeric.");
       }
     }
   }

   /**
    * Checks whether a character is a letter or digit (ascii) which are defined in the spec.
    */
   private static boolean isLetterOrDigit(char c) {
     return ('a' <= c && c <= 'z') || ('A' <= c && c <= 'Z') || ('0' <= c && c <= '9');
   }

   /**
    * 在Services MAP里面查找是否已有name的实例
    * */
   public S lookup(String name) {
     final S service = services.get(name);
     if (service != null) return service;
     throw new IllegalArgumentException("A SPI class of type "+clazz.getName()+" with name '"+name+"' does not exist. "+
      "You need to add the corresponding JAR file supporting this SPI to your classpath."+
      "The current classpath supports the following names: "+availableServices());
   }

   public Set<String> availableServices() {
     return services.keySet();
   }

   @Override
   public Iterator<S> iterator() {
     return services.values().iterator();
   }

   /**
    * Interface to support {@link NamedSPILoader#lookup(String)} by name.
    * <p>
    * Names must be all ascii alphanumeric, and less than 128 characters in length.
    */
   public static interface NamedSPI {
     String getName();
   }

 }

接下来看看Solr是怎么获取services的实例信息的

 package org.apache.lucene.util;

 /*
  * Licensed to the Apache Software Foundation (ASF) under one or more
  * contributor license agreements.  See the NOTICE file distributed with
  * this work for additional information regarding copyright ownership.
  * The ASF licenses this file to You under the Apache License, Version 2.0
  * (the "License"); you may not use this file except in compliance with
  * the License.  You may obtain a copy of the License at
  *
  *     http://www.apache.org/licenses/LICENSE-2.0
  *
  * Unless required by applicable law or agreed to in writing, software
  * distributed under the License is distributed on an "AS IS" BASIS,
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */

 import java.io.IOException;
 import java.io.InputStream;
 import java.io.BufferedReader;
 import java.io.InputStreamReader;
 import java.net.URL;
 import java.nio.charset.StandardCharsets;
 import java.util.ArrayList;
 import java.util.Collections;
 import java.util.Enumeration;
 import java.util.Iterator;
 import java.util.Locale;
 import java.util.NoSuchElementException;
 import java.util.ServiceConfigurationError;

 /**
  * Helper class for loading SPI classes from classpath (META-INF files).
  * This is a light impl of {@link java.util.ServiceLoader} but is guaranteed to
  * be bug-free regarding classpath order and does not instantiate or initialize
  * the classes found.
  *
  * @lucene.internal
  */
 public final class SPIClassIterator<S> implements Iterator<Class<? extends S>> {
   //service路径
   private static final String META_INF_SERVICES = "META-INF/services/";

   private final Class<S> clazz;
   private final ClassLoader loader;
   private final Enumeration<URL> profilesEnum;
   private Iterator<String> linesIterator;

   public static <S> SPIClassIterator<S> get(Class<S> clazz) {
     return new SPIClassIterator<>(clazz, Thread.currentThread().getContextClassLoader());
   }

   public static <S> SPIClassIterator<S> get(Class<S> clazz, ClassLoader loader) {
     return new SPIClassIterator<>(clazz, loader);
   }

   /** Utility method to check if some class loader is a (grand-)parent of or the same as another one.
    * This means the child will be able to load all classes from the parent, too. */
   public static boolean isParentClassLoader(final ClassLoader parent, ClassLoader child) {
     while (child != null) {
       if (child == parent) {
         return true;
       }
       child = child.getParent();
     }
     return false;
   }

   /**
    * 解析META-INF/services/clazz.getname文件
    * */
   private SPIClassIterator(Class<S> clazz, ClassLoader loader) {
     this.clazz = clazz;
     try {
       final String fullName = META_INF_SERVICES + clazz.getName();
       this.profilesEnum = (loader == null) ? ClassLoader.getSystemResources(fullName) : loader.getResources(fullName);
     } catch (IOException ioe) {
       throw new ServiceConfigurationError("Error loading SPI profiles for type " + clazz.getName() + " from classpath", ioe);
     }
     this.loader = (loader == null) ? ClassLoader.getSystemClassLoader() : loader;
     this.linesIterator = Collections.<String>emptySet().iterator();
   }

   /**
    * 获取META-INF/services/clazz.getname的clazz服务实例
    * */
   private boolean loadNextProfile() {
     ArrayList<String> lines = null;
     while (profilesEnum.hasMoreElements()) {
       if (lines != null) {
         lines.clear();
       } else {
         lines = new ArrayList<>();
       }
       final URL url = profilesEnum.nextElement();
       try {
         final InputStream in = url.openStream();
         IOException priorE = null;
         try {
           final BufferedReader reader = new BufferedReader(new InputStreamReader(in, StandardCharsets.UTF_8));
           String line;
           while ((line = reader.readLine()) != null) {
             final int pos = line.indexOf('#');
             if (pos >= 0) {
               line = line.substring(0, pos);
             }
             line = line.trim();
             if (line.length() > 0) {
               lines.add(line);
             }
           }
         } catch (IOException ioe) {
           priorE = ioe;
         } finally {
           IOUtils.closeWhileHandlingException(priorE, in);
         }
       } catch (IOException ioe) {
         throw new ServiceConfigurationError("Error loading SPI class list from URL: " + url, ioe);
       }
       if (!lines.isEmpty()) {
         this.linesIterator = lines.iterator();
         return true;
       }
     }
     return false;
   }

   @Override
   public boolean hasNext() {
     return linesIterator.hasNext() || loadNextProfile();
   }

   @Override
   public Class<? extends S> next() {
     // hasNext() implicitely loads the next profile, so it is essential to call this here!
     if (!hasNext()) {
       throw new NoSuchElementException();
     }
     assert linesIterator.hasNext();
     final String c = linesIterator.next();
     try {
       // don't initialize the class (pass false as 2nd parameter):
       return Class.forName(c, false, loader).asSubclass(clazz);
     } catch (ClassNotFoundException cnfe) {
       throw new ServiceConfigurationError(String.format(Locale.ROOT, "A SPI class of type %s with classname %s does not exist, "+
         "please fix the file '%s%1$s' in your classpath.", clazz.getName(), c, META_INF_SERVICES));
     }
   }

   @Override
   public void remove() {
     throw new UnsupportedOperationException();
   }

 }

由此可见SOLR SPI的流程是如下的：以Codec为例

1.SPIClassIterator获取所有META-INF/services/org.apache.lucene.codecs.Codec的实例类信息

2.NamedSPILoader实例化所有META-INF/services/org.apache.lucene.codecs.Codec的实例类，并放入services MAP里面

3.Codec默认为Lucene46，从services MAP获取Lucene46的实例类org.apache.lucene.codecs.lucene46.Lucene46Codec

 package org.apache.lucene.codecs.lucene46;

 /*
  * Licensed to the Apache Software Foundation (ASF) under one or more
  * contributor license agreements.  See the NOTICE file distributed with
  * this work for additional information regarding copyright ownership.
  * The ASF licenses this file to You under the Apache License, Version 2.0
  * (the "License"); you may not use this file except in compliance with
  * the License.  You may obtain a copy of the License at
  *
  *     http://www.apache.org/licenses/LICENSE-2.0
  *
  * Unless required by applicable law or agreed to in writing, software
  * distributed under the License is distributed on an "AS IS" BASIS,
  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  * See the License for the specific language governing permissions and
  * limitations under the License.
  */

 import org.apache.lucene.codecs.Codec;
 import org.apache.lucene.codecs.DocValuesFormat;
 import org.apache.lucene.codecs.FieldInfosFormat;
 import org.apache.lucene.codecs.FilterCodec;
 import org.apache.lucene.codecs.LiveDocsFormat;
 import org.apache.lucene.codecs.NormsFormat;
 import org.apache.lucene.codecs.PostingsFormat;
 import org.apache.lucene.codecs.SegmentInfoFormat;
 import org.apache.lucene.codecs.StoredFieldsFormat;
 import org.apache.lucene.codecs.TermVectorsFormat;
 import org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat;
 import org.apache.lucene.codecs.lucene41.Lucene41StoredFieldsFormat;
 import org.apache.lucene.codecs.lucene42.Lucene42NormsFormat;
 import org.apache.lucene.codecs.lucene42.Lucene42TermVectorsFormat;
 import org.apache.lucene.codecs.perfield.PerFieldDocValuesFormat;
 import org.apache.lucene.codecs.perfield.PerFieldPostingsFormat;

 /**
  * Implements the Lucene 4.6 index format, with configurable per-field postings
  * and docvalues formats.
  * <p>
  * If you want to reuse functionality of this codec in another codec, extend
  * {@link FilterCodec}.
  *
  * @see org.apache.lucene.codecs.lucene46 package documentation for file format details.
  * @lucene.experimental
  */
 // NOTE: if we make largish changes in a minor release, easier to just make Lucene46Codec or whatever
 // if they are backwards compatible or smallish we can probably do the backwards in the postingsreader
 // (it writes a minor version, etc).
 public class Lucene46Codec extends Codec {
   private final StoredFieldsFormat fieldsFormat = new Lucene41StoredFieldsFormat();
   private final TermVectorsFormat vectorsFormat = new Lucene42TermVectorsFormat();
   private final FieldInfosFormat fieldInfosFormat = new Lucene46FieldInfosFormat();
   private final SegmentInfoFormat segmentInfosFormat = new Lucene46SegmentInfoFormat();
   private final LiveDocsFormat liveDocsFormat = new Lucene40LiveDocsFormat();

   private final PostingsFormat postingsFormat = new PerFieldPostingsFormat() {
     @Override
     public PostingsFormat getPostingsFormatForField(String field) {
       return Lucene46Codec.this.getPostingsFormatForField(field);
     }
   };

   private final DocValuesFormat docValuesFormat = new PerFieldDocValuesFormat() {
     @Override
     public DocValuesFormat getDocValuesFormatForField(String field) {
       return Lucene46Codec.this.getDocValuesFormatForField(field);
     }
   };

   /** Sole constructor. */
   public Lucene46Codec() {
     super("Lucene46");
   }

   @Override
   public final StoredFieldsFormat storedFieldsFormat() {
     return fieldsFormat;
   }

   @Override
   public final TermVectorsFormat termVectorsFormat() {
     return vectorsFormat;
   }

   @Override
   public final PostingsFormat postingsFormat() {
     return postingsFormat;
   }

   @Override
   public final FieldInfosFormat fieldInfosFormat() {
     return fieldInfosFormat;
   }

   @Override
   public final SegmentInfoFormat segmentInfoFormat() {
     return segmentInfosFormat;
   }

   @Override
   public final LiveDocsFormat liveDocsFormat() {
     return liveDocsFormat;
   }

   /** Returns the postings format that should be used for writing
    *  new segments of <code>field</code>.
    *
    *  The default implementation always returns "Lucene41"
    */
   public PostingsFormat getPostingsFormatForField(String field) {
     return defaultFormat;
   }

   /** Returns the docvalues format that should be used for writing
    *  new segments of <code>field</code>.
    *
    *  The default implementation always returns "Lucene45"
    */
   public DocValuesFormat getDocValuesFormatForField(String field) {
     return defaultDVFormat;
   }

   @Override
   public final DocValuesFormat docValuesFormat() {
     return docValuesFormat;
   }

   private final PostingsFormat defaultFormat = PostingsFormat.forName("Lucene41");
   private final DocValuesFormat defaultDVFormat = DocValuesFormat.forName("Lucene45");

   private final NormsFormat normsFormat = new Lucene42NormsFormat();

   @Override
   public final NormsFormat normsFormat() {
     return normsFormat;
   }
 }