[FIXED] Maven build [WARNING] we have a duplicate class

Issue

Anybody has any idea what happened to my maven build? I am getting a lot of duplicate warnings.

[WARNING] We have a duplicate org/apache/commons/logging/impl/LogFactoryImpl$1.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
[WARNING] We have a duplicate org/apache/commons/logging/impl/LogFactoryImpl.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
[WARNING] We have a duplicate org/apache/commons/logging/impl/NoOpLog.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
[WARNING] We have a duplicate org/apache/commons/logging/impl/SimpleLog$1.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
[WARNING] We have a duplicate org/apache/commons/logging/impl/SimpleLog.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar
[WARNING] We have a duplicate org/apache/commons/logging/impl/Jdk14Logger.class in /home/shengjie/.m2/repository/commons-logging/commons-logging-api/1.0.4/commons-logging-api-1.0.4.jar

I've looked into my local m2 repo, I have two classes there in commons-logging-api jar, LogFactoryImpl.class and LogFactoryImpl$1.class. Same as all the classes mentioned in the warnings.

One thing to mention is that I am using shade plugin in my pom.xml.

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-shade-plugin</artifactId>
            <version>1.4</version>
            <configuration>
                <createDependencyReducedPom>true</createDependencyReducedPom>
                <filters>
                    <filter>
                        <artifact>*:*</artifact>
                        <excludes>
                            <exclude>META-INF/*.SF</exclude>
                            <exclude>META-INF/*.DSA</exclude>
                            <exclude>META-INF/*.RSA</exclude>
                        </excludes>
                    </filter>
                </filters>
            </configuration>
            <executions>
                <execution>
                    <phase>package</phase>
                    <goals>
                        <goal>shade</goal>
                    </goals>
                    <configuration>
                        <transformers>
                            <transformer
                                implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer" />
                            <transformer
                                implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">
                                <mainClass>com.~~~~black out my own main class here~~~~~</mainClass>
                            </transformer>
                        </transformers>
                    </configuration>
                </execution>
            </executions>
        </plugin>

I noticed that the dependency tree looks like as below

[INFO] +- org.apache.cxf:cxf-bundle-jaxrs:jar:2.5.1:compile
[INFO] |  \- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] \- org.apache.hadoop.hive:hive-jdbc:jar:0.7.1-cdh3u3:compile
[INFO]    \- org.apache.hadoop.hive:hive-common:jar:0.7.1-cdh3u3:compile
[INFO]       \- commons-logging:commons-logging-api:jar:1.0.4:compile

and commons-logging.jar and commons-logging-api.jar both have org/apache/commons/logging/LogFactory.class.

somehow Shad plugin is trying to squeeze them in to a big fat jar at the end. then the warning is showing up. It's been said this is ignorable warning. But I am a bit worried, How does the application know what is the exact class should be used if there are two duplicated class with the same name?

Solution

Take a look at the "Dependency Exclusions" section in the Maven doc.

In your provided example, I'll exclude the commons-logging:commons-logging-api:jar:1.0.4:compile dependency from org.apache.hadoop.hive:hive-common:jar:0.7.1-cdh3u3:compile. In your pom.xml :

    <dependency>
        <groupId>org.apache.hadoop.hive</groupId>
        <artifactId>hive-common:jar</artifactId>
        <version>0.7.1-cdh3u3</version>
        <exclusions>
            <exclusion>
                <groupId>commons-logging</groupId>
                <artifactId>commons-logging-api</artifactId>
            </exclusion>
        </exclusions>
    </dependency>

Answered By - ndeverge
Answer Checked By - Senaida (JavaFixing Volunteer)