Running Solr with Maven

Solr is an open source search server which is built by using the indexing and search capabilities of Lucene Core, and it can be used for implementing scalable search engines with almost any programming language.

Even though Solr has many advantages, setting up a a development environment is not one of them. This blog entry describes how we can run Solr by using Maven and ensure that each developer uses the same configuration, schema and Solr version.

The requirements of our Maven build are following:

• The properties of our Maven build must be read from an external property file. The only exception to this rule is that the version numbers of the dependencies are declared in our POM file.
• The build process must copy the Solr configuration files to the correct directory when our Solr instance is started.
• The build process must clean up the configuration files when a developer executes mvn cleancommand at command prompt.
• It must be possible to start our Solr instance by using the Jetty Maven plugin.

We can fulfil these requirements by following these steps:

1. Create a POM file.
2. Get the required dependencies.
3. Get the Solr configuration files.
4. Create the properties file which contain the properties used in our Maven build.
5. Edit the solr.xml file.
6. Configure the Properties Maven plugin.
7. Configure the Copy Maven plugin.
8. Configure the Jetty Maven plugin.

These steps are described with more details in the following.

Creating the POM file

First, We have to create a POM file for a web application project. The skeleton of our POM file looks as follows:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>net.petrikainulainen.maven</groupId>
    <artifactId>running-solr-with-maven</artifactId>
    <packaging>war</packaging>
    <version>0.1</version>
    
    <profiles>
        <!-- Add profile configuration here -->
    </profiles>
    <dependencies>
        <!-- Add dependencies here -->
    </dependencies>
    <build>
        <finalName>solr</finalName>
        <!-- Add filter configuration here -->
        <!-- Add resources configuration here -->
        <plugins>
            <!-- Add plugin configuration here -->
        </plugins>
    </build>
</project>

Getting the Required Dependencies

We need to configure the following dependencies in our pom.xml file:

• SLF4J
• SLF4J interceptors for both java.util.logging (JUL) and java.commons.logging (JCL) logging frameworks.
• SLF4J Log4j 1.2.x binding
• Log4j
• Solr 4.3.0 (war)

Note: The reason why we have to configure the logging jars as dependencies is that the logging setup of Solr was changed when Solr 4.3.0 was released.

In other words, we have to add the following dependency declarations to the dependenciessection of our POM file:

<!-- SLF4J -->
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-api</artifactId>
    <version>1.7.5</version>
</dependency>
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>jcl-over-slf4j</artifactId>
    <version>1.7.5</version>
</dependency>
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>jul-to-slf4j</artifactId>
    <version>1.7.5</version>
</dependency>
<dependency>
    <groupId>org.slf4j</groupId>
    <artifactId>slf4j-log4j12</artifactId>
    <version>1.7.5</version>
</dependency>
<!-- Log4j -->
<dependency>
    <groupId>log4j</groupId>
    <artifactId>log4j</artifactId>
    <version>1.2.17</version>
</dependency>
<!-- Solr 4.3.0 -->
<dependency>
    <groupId>org.apache.solr</groupId>
    <artifactId>solr</artifactId>
    <version>4.3.0</version>
    <type>war</type>
</dependency>

Getting the Solr Configuration Files

We can get the Solr configuration files by following these steps:

1. Download the binary distribution of Solr 4.3.0.
2. Extract the downloaded package to the desired directory.
3. Go the root directory of the extracted Solr binary distribution.
4. Copy the following files from the directory example/solr/collection1/conf to the directorysrc/main/config: admin-extra.html, admin-extra-menu.menu-bottom.html, admin-extra.menu-top.hml, currency.xml, elevate.xml, mapping-FoldToASCII.txt, mapping-ISOLatin1Accent.txt, protwords.xml, schema.xml, scripts.conf, solrconfig.xml, spellings.txt, stopwords.txt, synonyms.txt and update-script.js.
5. Copy the language specific configuration files found from directoryexample/solr/collection1/conf/lang to the directry src/main/config/lang.
6. Copy the Velocity macros and other files found from the directoryexample/solr/collection1/conf/velocity to the directry src/main/config/velocity.
7. Copy the XSL style sheets found from the directory example/solr/collection1/conf/xslt to the directry src/main/config/xslt.
8. Copy the solr.xml file from the directory exaple/solr/collection1 to the directorysrc/main/resources.
9. Create a directory src/main/webapp/WEB-INF. This directory is required so that the Solr instance can be started.

We have now successfully obtained the required files and are ready to move on to the next phase.

Creating the Properties File

Our next is the to create the properties file that is used in our Maven build and add the required build profile configuration to our POM file. Lets move on and find out how this is done.

First, we have create the properties file which is used in our Maven build. We can do this by following these steps:

1. Create directory profiles/dev to the root directory of our Maven project.
2. Create a properties file config.properties to the profiles/dev directory.

Our properties file has three properties which are described in the following:

• The solr.detault.core.directory property states the value of the default core directory. This is a directory which is created under the home directory of our Solr instance. This directory stores the configuration of our Solr instance and its data.
• The solr.default.core.name property states the name of the default core.
• The solr.solr.home property states the home directory of our Solr installation.

The content of the config.properties file looks as follows:

#SOLR PROPERTIES
#Configures the directory used to store the data and configuration of the Solr default core
solr.default.core.directory=todo
#Configures the name of the Solr default core.
solr.default.core.name=todo

#SYSTEM PROPERTIES
#Configures the home directory of Solr. Set the preferred directory path here.
solr.solr.home=

Second, we must configure the build profiles of our Maven build and use filtering to replace replace the variables included in our resources. We can do this by following these steps:

1. Create a single profile called dev and ensure that it is the default profile of our build.
2. Declare a property called build.profile.id and set its value to ‘dev’.
3. Create a filter that reads the profile specific configuration file and replaces the variables found from our resources with the actual property values.

We can finish steps one and two by adding the following profile declaration to our POM file:

<profile>
    <id>dev</id>
    <activation>
        <activeByDefault>true</activeByDefault>
    </activation>
    <properties>
        <build.profile.id>dev</build.profile.id>
    </properties>
</profile>

We can finish step three by adding the following XML to the build section of our POM file:

<filters>
    <filter>${project.basedir}/profiles/${build.profile.id}/config.properties</filter>
</filters>
<resources>
    <resource>
        <filtering>true</filtering>
        <directory>src/main/resources</directory>
    </resource>
</resources>

Editing the solr.xml File

Because we use a profile specific configuration file to configure the name and the instance directory of the Solr default core, we have to make changes to the solr.xml file. These changes are described in the following:

1. The value of the solr.default.core.name property must be set as the value of thedefaultCoreNameAttribute attribute of the cores element.
2. The value of the solr.default.core.name property must be set as the value of the nameattribute of the core element.
3. The value of the solr.default.core.directory property must be set as the value of theinstanceDir attribute of the core element.

The content of the solr.xml file looks as follows:

<solr persistent="true">
  <cores adminPath="/admin/cores" defaultCoreName="${solr.default.core.name}" host="${host:}" hostPort="${jetty.port:}" hostContext="${hostContext:}" zkClientTimeout="${zkClientTimeout:15000}">
    <core name="${solr.default.core.name}" instanceDir="${solr.default.core.directory}" />
  </cores>
</solr>

Configuring the Properties Maven Plugin

Because we want that all property values used in our POM file are read from an external properties file, we have to use a plugin called the Properties Maven plugin. We can configure this plugin by following these steps:

1. Ensure that the properties are read from the profile specific configuration file.
2. Create an execution that runs the read-project-properties goal of the Properties Maven plugin in the initialize phase of the Maven default lifecycle.
3. Create an execution that runs the read-project properties goal of the Properties Maven plugin in the pre-clean phase of the Maven clean lifecycle.

The configuration of the Properties Maven plugin looks as follows:

<plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>properties-maven-plugin</artifactId>
    <version>1.0-alpha-2</version>
    <configuration>
        <files>
            <!-- Properties are read from profile specific property file -->
            <file>${project.basedir}/profiles/${build.profile.id}/config.properties</file>
        </files>
    </configuration>
    <executions>
        <!-- Load properties for the default lifecycle -->
        <execution>
            <id>default-lifecycle-properties</id>
            <phase>initialize</phase>
            <goals>
                <goal>read-project-properties</goal>
            </goals>
        </execution>
        <!-- Load properties for the clean lifecycle -->
        <execution>
            <id>clean-lifecycle-properties</id>
            <phase>pre-clean</phase>
            <goals>
                <goal>read-project-properties</goal>
            </goals>
        </execution>
    </executions>
</plugin>

Configuring the Copy Maven Plugin

We will use the Copy Maven plugin for two purposes:

1. We copy the Solr configuration files to the correct directory when we start our Solr instance.
2. We delete the Solr configuration files when we execute command mvn clean at command prompt.

We can get started by adding the following XML to the plugins section of our POM file:

<plugin>
    <groupId>com.github.goldin</groupId>
    <artifactId>copy-maven-plugin</artifactId>
    <version>0.2.5</version>
    <executions>
        <!-- Add executions here -->
    </executions>
</plugin>

Lets move on and find out how we can configure the Copy Maven plugin to copy and delete the Solr configuration files.

Copying Solr Configuration Files

We can copy the Solr configuration files by following these steps:

1. Create an execution which runs the copy goal of Copy Maven plugin in the compile phase of the Maven default lifecycle.
2. Copy the solr.xml file the home directory of our Solr instance. Ensure that the properties filtering is applied to file when it is copied.
3. Copy the files found from the src/main/config directory to thesolr.solr.home/solr.default.core.directory/conf directory.
4. Copy the language specific configuration files found from the src/main/config/lang directory to the solr.solr.home/solr.detault.core.directory/conf/lang directory.
5. Copy the Velocity macros and other files found from the src/main/config/velocity directory to the solr.solr.home/solr.detault.core.directory/conf/velocity directory.
6. Copy the XSL style sheets found from the src/main/config/xslt directory to thesolr.solr.home/solr.detault.core.directory/conf/xslt directory.

The configuration of our execution looks as follows:

<execution>
    <id>copy-solr-config</id>
    <phase>compile</phase>
    <goals>
        <goal>copy</goal>
    </goals>
    <configuration>
        <resources>
            <!--
           Copy solr.xml to correct directory and applies properties
           filtering to it.
           -->
            <resource>
                <directory>${project.basedir}/src/main/resources</directory>
                <filtering>true</filtering>
                <targetPath>${solr.solr.home}</targetPath>
                <includes>
                    <include>solr.xml</include>
                </includes>
            </resource>
            <!-- Copy configuration files -->
            <resource>
                <directory>${project.basedir}/src/main/config</directory>
                <targetPath>${solr.solr.home}/${solr.default.core.directory}/conf</targetPath>
                <excludes>
                    <exclude>lang</exclude>
                    <exclude>velocity</exclude>
                    <exclude>xslt</exclude>
                </excludes>
            </resource>
            <!-- Copy language specific configuration files -->
            <resource>
                <directory>${project.basedir}/src/main/config/lang</directory>
                <targetPath>${solr.solr.home}/${solr.default.core.directory}/conf/lang</targetPath>
            </resource>
            <!-- Copy Velocity macros and other files -->
            <resource>
                <directory>${project.basedir}/src/main/config/velocity</directory>
                <targetPath>${solr.solr.home}/${solr.default.core.directory}/conf/velocity</targetPath>
            </resource>
            <!-- Copy XSL style sheets -->
            <resource>
                <directory>${project.basedir}/src/main/config/xslt</directory>
                <targetPath>${solr.solr.home}/${solr.default.core.directory}/conf/xslt</targetPath>
            </resource>
        </resources>
    </configuration>
</execution>

Deleting Solr Configuration Files

We can delete the Solr configuration files by following these steps:

1. Create an execution which runs the copy goal of the Copy Maven plugin in the clean lifecycle phase.
2. Ensure that build does not fail if the directories are not found.
3. Delete the overlays directory which is created to the root directory of our Maven project.
4. Delete the solr.xml file found from the home directory of our Solr instance.
5. Delete the conf directory found from the solr.solr.home/solr.default.core.directory directory.

The configuration of our execution looks as follows:

<execution>
    <id>clean-solr</id>
    <phase>clean</phase>
    <goals>
        <goal>copy</goal>
    </goals>
    <configuration>
        <failIfNotFound>false</failIfNotFound>
        <resources>
            <!-- Clean the overlays directory from the project root directory -->
            <resource>
                <clean>true</clean>
                <cleanEmptyDirectories>true</cleanEmptyDirectories>
                <directory>${project.basedir}/overlays</directory>
                <includes>
                    <include>**/**</include>
                </includes>
            </resource>
            <!-- Remove the solr.xml file -->
            <resource>
                <clean>true</clean>
                <directory>${solr.solr.home}</directory>
                <includes>
                    <include>solr.xml</include>
                </includes>
            </resource>
            <!-- Remove the conf directory -->
            <resource>
                <clean>true</clean>
                <cleanEmptyDirectories>true</cleanEmptyDirectories>
                <directory>${solr.solr.home}/${solr.default.core.directory}</directory>
                <includes>
                    <include>conf</include>
                </includes>
            </resource>
        </resources>
    </configuration>
</execution>

Configuring the Jetty Maven Plugin

We can configure the Jetty Maven plugin to run our Solr instance by following these steps:

1. Configure Jetty to listen the port 8983.
2. Ensure that system properties are read from the profile specific configuration file. This property file contains a property called solr.solr.home which specifies the home directory of our Solr instance.
3. Specify that the context path of our application is /solr.

The configuration of the Jetty Maven plugin looks as follows:

<plugin>
    <groupId>org.mortbay.jetty</groupId>
    <artifactId>jetty-maven-plugin</artifactId>
    <version>8.1.8.v20121106</version>
    <configuration>
        <stopPort>9966</stopPort>
        <stopKey>stop</stopKey>
        <connectors>
            <!-- Listen to port 8983 -->
            <connector implementation="org.eclipse.jetty.server.nio.SelectChannelConnector">
                <port>8983</port>
                <maxIdleTime>60000</maxIdleTime>
            </connector>
        </connectors>
        <!-- Read system properties from profile specific configuration file -->
        <systemPropertiesFile>${project.basedir}/profiles/${build.profile.id}/config.properties</systemPropertiesFile>
        <webApp>
            <contextPath>/solr</contextPath>
        </webApp>
    </configuration>
</plugin>

Running Solr

We have now created a Maven build which can be used to run Solr in a development environment. We have got two options for starting our Solr instance:

• We can execute mvn jetty:run command at command prompt.
• We can execute mvn jetty:run-war command at command prompt.

After we have started Solr, we can access its admin interface by using the following url address:http://localhost:8983/solr.

The example application is available at Github. This example uses a custom schema because I plan to use it in my Spring Data Solr tutorial. The original example schema is found from the etcdirectory.