安全地管理 JAR 依赖项Safely manage jar dependencies
安装在 HDInsight 群集上的组件依赖于第三方库。Components installed on HDInsight clusters have dependencies on third-party libraries. 通常,这些内置组件引用特定版本的通用模块(例如 Guava)。Usually, a specific version of common modules like Guava is referenced by these built-in components. 连同依赖项一起提交某个应用程序时,可能会导致同一模块的不同版本之间发生冲突。When you submit an application with its dependencies, it can cause a conflict between different versions of the same module. 如果先在类路径中引用某个组件版本,内置组件可能会因版本不兼容而引发异常。If the component version that you reference in the classpath first, built-in components may throw exceptions because of version incompatibility. 但是,如果内置组件先在类路径注入其依赖项,则应用程序可能会引发 NoSuchMethod
之类的错误。However, if built-in components inject their dependencies to the classpath first, your application may throw errors like NoSuchMethod
.
为了避免版本冲突,请考虑分装应用程序依赖项。To avoid version conflict, consider shading your application dependencies.
包分装是什么意思?What does package shading mean?
“分装”是包含和重命名依赖项的一种方式。Shading provides a way to include and rename dependencies. 它将类重新定位,并重写受影响的字节码和资源,以创建依赖项的专用副本。It relocates the classes and rewrites affected bytecode and resources to create a private copy of your dependencies.
如何将包分装?How to shade a package?
使用 uber-jarUse uber-jar
uber-jar 是单个 jar 文件,其中包含应用程序 jar 及其依赖项。Uber-jar is a single jar file that contains both the application jar and its dependencies. uber-jar 中的依赖项默认不会分装。The dependencies in Uber-jar are by-default not shaded. 在某些情况下,如果其他组件或应用程序引用这些库的不同版本,则可能会导致版本冲突。In some cases, this may introduce version conflict if other components or applications reference a different version of those libraries. 为了避免这种情况,可以生成一个 uber-jar 文件并在其中包含一部分(或全部)已分装的依赖项。To avoid this, you can build an Uber-Jar file with some (or all) of the dependencies shaded.
使用 Maven 将包分装Shade package using Maven
Maven 可以生成以 Java 和 Scala 编写的应用程序。Maven can build applications written both in Java and Scala. Maven-shade-plugin 可帮助你轻松创建分装的 uber-jar。Maven-shade-plugin can help you create a shaded uber-jar easily.
以下示例演示一个 pom.xml
文件,该文件已更新为使用 maven-shade-plugin 将包分装。The example below shows a file pom.xml
which has been updated to shade a package using maven-shade-plugin. XML 节 <relocation>…</relocation>
通过移动相应的 JAR 文件条目并重写受影响的字节码,将类从包 com.google.guava
移到包 com.google.shaded.guava
中。The XML section <relocation>…</relocation>
moves classes from package com.google.guava
into package com.google.shaded.guava
by moving the corresponding JAR file entries and rewriting the affected bytecode.
更改 pom.xml
后,可以执行 mvn package
来生成分装的 uber-jar。After changing pom.xml
, you can execute mvn package
to build the shaded uber-jar.
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>3.2.1</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<relocations>
<relocation>
<pattern>com.google.guava</pattern>
<shadedPattern>com.google.shaded.guava</shadedPattern>
</relocation>
</relocations>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
使用 SBT 将包分装Shade package using SBT
SBT 也是适用于 Scala 和 Java 的生成工具。SBT is also a build tool for Scala and Java. SBT 没有类似于 maven-shade-plugin 的分装插件。SBT doesn't have a shade plugin like maven-shade-plugin. 可以修改 build.sbt
文件将包分装。You can modify build.sbt
file to shade packages.
例如,若要分装 com.google.guava
,可将以下命令添加到 build.sbt
文件:For example, to shade com.google.guava
, you can add the below command to the build.sbt
file:
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.google.guava" -> "com.google.shaded.guava.@1").inAll
)
然后,可以运行 sbt clean
和 sbt assembly
来生成分装的 jar 文件。Then you can run sbt clean
and sbt assembly
to build the shaded jar file.