Skip to content

Latest commit

 

History

History
 
 

flink

README.md

Flink Interpreter

This is the doc for Zeppelin developers who want to work on flink interpreter.

Development Guide

Project Structure

Flink interpreter is more complex than other interpreter (such as jdbc, shell). Currently it has following 8 modules

  • flink-shims
  • flink1.10-shims
  • flink1.11-shims
  • flink1.12-shims
  • flink1.13-shims
  • flink-scala-parent
  • flink-scala-2.11
  • flink-scala-2.12

The first 5 modules are to adapt different flink versions because there're some api changes between different versions of flink. flink-shims is parent module for other shims modules. At runtime Flink interpreter will load the FlinkShims based on the current flink versions (See FlinkShims#loadShims).

The remaining 3 modules are to adapt different scala versions (Apache Flink supports 2 scala versions: 2.11 & 2.12). flink-scala-parent is a parent module for flink-scala-2.11 and flink-scala-2.12. It contains common code for both flink-scala-2.11 and flink-scala-2.12. There's symlink folder flink-scala-parent under flink-scala-2.11 and flink-scala-2.12. When you run maven command to build flink interpreter, the source code in flink-scala-parent won't be compiled directly, instead they will be compiled against different scala versions when building flink-scala-2.11 & flink-scala-2.12. (See build-helper-maven-plugin in pom.xml) Both flink-scala-2.11 and flink-scala-2.12 build a flink interpreter jar and FlinkInterpreterLauncher in zeppelin-plugins/launcher/flink will choose the right jar based on the scala version of flink.

Work in IDE

Because of the complex project structure of flink interpreter, we need to do more configuration to make it work in IDE. Here we take Intellij as an example (other IDE should be similar).

The key point is that we can only make flink interpreter work with one scala version at the same time in IDE. So we have to disable the other module when working with one specific scala version module.

Make it work with scala-2.11

  1. Exclude the source code folder (java/scala) of flink-scala-parent (Right click these folder -> Mark directory As -> Excluded)
  2. Include the source code folder (java/scala) of flink/flink-scala-2.11/flink-scala-parent (Right click these folder -> Mark directory As -> Source root)

Make it work with scala-2.12

  1. Exclude the source code folder (java/scala) of flink-scala-parent (Right click these folder -> Mark directory As -> Excluded)
  2. Include the source code folder (java/scala) of flink/flink-scala-2.12/flink-scala-parent (Right click these folder -> Mark directory As -> Source root)

How to run unit test in IDE

Take FlinkInterpreterTest as an example, you need to specify environment variables FLINK_HOME, FLINK_CONF_DIR, ZEPPELIN_HOME. See maven-surefire-plugin in pom.xml for more details