feat(mlkit): Migrate Face, Barcode and Text Recognition to Firebase M…

…LKit (iOS - text, Android - all) (react-native-camera#2075) * switch to firebase mlkit for text recognition * migrate android to MLKit migrate Text detection android to MLKit update gradle of example app Update build.gradle separate gms vision facedetector to general flavor migrate faceDetector to mlkit and fix incorrect bounds due to padding migrate barCode detector to mlkit update android instructions in readme safe face implementation move gms to generalImplementation * add mlkit example setup android of mlkit example setup ios of mlkit example fix typo in readme update example project readme * amend mlkit migration to include raw data add barcode detection in basic and mlkit examples * fix duplicate bridgeDidBackground method BREAKING CHANGE: We migrated to MLKit instead of Google Mobile Vision
xaphod · Mar 11, 2019 · 0ab570a · 0ab570a
1 parent 1a89d79
commit 0ab570a
Show file tree

Hide file tree

Showing 115 changed files with 14,512 additions and 1,346 deletions.
diff --git a/.gitignore b/.gitignore
@@ -45,4 +45,7 @@ android/keystores/debug.keystore
 package-json.lock
 
 # vscode
-.vscode
+.vscode
+examples/mlkit/android/app/google-services.json
+examples/mlkit/ios/Pods
+examples/mlkit/ios/mlkit/GoogleService-Info.plist
diff --git a/README.md b/README.md
@@ -256,7 +256,31 @@ pod 'react-native-camera', path: '../node_modules/react-native-camera', subspecs
   'TextDetector'
 ]
 ```
-*Note:* Text recognition is available only via CocoaPods Path. If you have issues with duplicate symbols you will need to enable dead code stripping option in your Xcode (Target > Build Settings > search for "Dead code stripping") see: https://github.com/firebase/quickstart-ios/issues/487#issuecomment-415313053.
+
+###### Additional steps for Text Recognition
+Text recognition for iOS uses Firebase MLKit which requires setting up Firebase project for your app.
+If you have not already added Firebase to your app, please follow the steps described in [getting started guide](https://firebase.google.com/docs/ios/setup).
+In short, you would need to
+1. Register your app in Firebase console.
+2. Download `GoogleService-Info.plist` and add it to your project
+3. Add `pod 'Firebase/Core'` to your podfile
+4. In your `AppDelegate.m` file add the following lines:
+```objective-c
+#import <Firebase.h> // <--- add this
+...
+
+- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions
+{
+  [FIRApp configure]; // <--- add this
+  ...
+}
+```
+
+Add pods `Firebase/MLVision` and `Firebase/MLVisionTextModel`
+
+*Note:* Text recognition is currently available only via CocoaPods Path. 
+- If you have issues with duplicate symbols you will need to enable dead code stripping option in your Xcode (Target > Build Settings > search for "Dead code stripping") [see here](https://github.com/firebase/quickstart-ios/issues/487#issuecomment-415313053).
+- If you are using `pod Firebase/Core` with a version set below 5.13 you might want to add `pod 'GoogleAppMeasurement', '~> 5.3.0'` to your podfile
 
 ###### Non-CocoaPods Path
 1. Download:
@@ -291,23 +315,42 @@ Google Symbol Utilities: https://www.gstatic.com/cpdc/dbffca986f6337f8-GoogleSym
 	project(':react-native-camera').projectDir = new File(rootProject.projectDir, 	'../node_modules/react-native-camera/android')
 	```
 
-4. Insert the following lines inside the dependencies block in `android/app/build.gradle`:
+4. Insert the following lines in `android/app/build.gradle`:
+
+  inside the dependencies block:
 
 	```gradle
-    compile (project(':react-native-camera')) {
-        exclude group: "com.google.android.gms"
-        compile 'com.android.support:exifinterface:25.+'
-        compile ('com.google.android.gms:play-services-vision:12.0.1') {
-            force = true
-        }
-    }
+    implementation project(':react-native-camera')
 	```
 
-  > You may need to use different exifinterface versions, e.g. `27.+` instead of `25.+`.
+  inside defaultConfig block insert either:
+
+  ```gradle
+  android {
+    ...
+    defaultConfig {
+      ...
+      missingDimensionStrategy 'react-native-camera', 'general' <-- insert this line
+    }
+  }
+  ```
+
+  or, if using MLKit for text/face/barcode recognition:
+
+  ```gradle
+  android {
+    ...
+    defaultConfig {
+      ...
+      missingDimensionStrategy 'react-native-camera', 'mlkit' <-- insert this line
+    }
+  }
+  ```
+
 
 5. Declare the permissions in your Android Manifest (required for `video recording` feature)
 
-  ```java
+  ```xml
   <uses-permission android:name="android.permission.RECORD_AUDIO"/>
   <uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
   <uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
@@ -323,6 +366,38 @@ allprojects {
 }
 ```
 
+7. Additional steps for using MLKit for text/face/barcode recognition
+
+    7.1. Using Firebase MLKit requires seting up Firebase project for your app. If you have not already added Firebase to your app, please follow the steps described in [getting started guide](https://firebase.google.com/docs/android/setup).
+    In short, you would need to
+    - Register your app in Firebase console.
+    - Download google-services.json and place it in `android/app/`
+    - add the folowing to project level build.gradle:
+    ```gradle
+        buildscript {
+          dependencies {
+          // Add this line
+          classpath 'com.google.gms:google-services:4.0.1' <-- you might want to use different version
+          }
+        }
+    ```
+    - add to the bottom of `android/app/build.gradle` file
+    ```gradle 
+    apply plugin: com.google.gms.google-services'
+    ```
+
+    7.2. Configure your app to automatically download the ML model to the device after your app is installed from the Play Store. If you do not enable install-time model downloads, the model will be downloaded the first time you run the on-device detector. Requests you make before the download has completed will produce no results. 
+
+    ```xml
+    <application ...>
+    ...
+      <meta-data
+          android:name="com.google.firebase.ml.vision.DEPENDENCIES"
+          android:value="ocr" />
+      <!-- To use multiple models, list all needed models: android:value="ocr, face, barcode" -->
+    </application>
+    ```
+
 The current Android library defaults to the below values for the Google SDK and Libraries,
 
 ```gradle
@@ -359,45 +434,88 @@ The above settings in the ReactNative project over-rides the values present in t
 module. For your reference below is the `android/build.gradle` file of the module.
 
 ```gradle
+def safeExtGet(prop, fallback) {
+    rootProject.ext.has(prop) ? rootProject.ext.get(prop) : fallback
+}
+
 buildscript {
-...
+  repositories {
+    google()
+    maven {
+      url 'https://maven.google.com'
+    }
+    jcenter()
+  }
 
-def DEFAULT_COMPILE_SDK_VERSION             = 26
-def DEFAULT_BUILD_TOOLS_VERSION             = "26.0.2"
-def DEFAULT_TARGET_SDK_VERSION              = 26
-def DEFAULT_GOOGLE_PLAY_SERVICES_VERSION    = "12.0.1"
-def DEFAULT_SUPPORT_LIBRARY_VERSION         = "27.1.0"
+  dependencies {
+    classpath 'com.android.tools.build:gradle:3.0.1'
+  }
+}
+
+apply plugin: 'com.android.library'
 
 android {
-  compileSdkVersion rootProject.hasProperty('compileSdkVersion') ? rootProject.compileSdkVersion : DEFAULT_COMPILE_SDK_VERSION
-  buildToolsVersion rootProject.hasProperty('buildToolsVersion') ? rootProject.buildToolsVersion : DEFAULT_BUILD_TOOLS_VERSION
+  compileSdkVersion safeExtGet('compileSdkVersion', 26)
+  buildToolsVersion safeExtGet('buildToolsVersion', '26.0.2')
 
   defaultConfig {
-    minSdkVersion 16
-    targetSdkVersion rootProject.hasProperty('targetSdkVersion') ? rootProject.targetSdkVersion : DEFAULT_TARGET_SDK_VERSION
+    minSdkVersion safeExtGet('minSdkVersion', 16)
+    targetSdkVersion safeExtGet('targetSdkVersion', 26)
+  }
 
-    versionCode 1
-    versionName "1.0.0"
+  flavorDimensions "react-native-camera"
+
+  productFlavors {
+    general {
+      dimension "react-native-camera"
+    }
+    mlkit {
+      dimension "react-native-camera"
+    }
   }
+
+  sourceSets {
+    main {
+      java.srcDirs = ['src/main/java']
+    }
+    general {
+      java.srcDirs = ['src/general/java']
+    }
+    mlkit {
+      java.srcDirs = ['src/mlkit/java']
+    }
+  }
+
   lintOptions {
     abortOnError false
     warning 'InvalidPackage'
   }
 }
 
-...
+repositories {
+  google()
+  mavenCentral()
+  maven {
+   url 'https://maven.google.com'
+  }
+  maven { url "https://jitpack.io" }
+  maven {
+    // All of React Native (JS, Obj-C sources, Android binaries) is installed from npm
+    url "$rootDir/../node_modules/react-native/android"
+  }
+}
 
 dependencies {
-  def googlePlayServicesVersion = rootProject.hasProperty('googlePlayServicesVersion')  ? rootProject.googlePlayServicesVersion : DEFAULT_GOOGLE_PLAY_SERVICES_VERSION
-  def supportLibVersion = rootProject.hasProperty('supportLibVersion')  ? rootProject.supportLibVersion : DEFAULT_SUPPORT_LIBRARY_VERSION
-
-  compile 'com.facebook.react:react-native:+'
-  compile "com.google.zxing:core:3.2.1"
-  compile "com.drewnoakes:metadata-extractor:2.9.1"
-  compile 'com.google.android.gms:play-services-vision:$googlePlayServicesVersion'
-  compile 'com.android.support:exifinterface:$supportLibVersion'
-
-  compile 'com.github.react-native-community:cameraview:cc47bb28ed2fc54a8c56a4ce9ce53edd1f0af3a5'
+  compileOnly 'com.facebook.react:react-native:+'
+  compileOnly 'com.facebook.infer.annotation:infer-annotation:+'
+  implementation "com.google.zxing:core:3.3.0"
+  implementation "com.drewnoakes:metadata-extractor:2.9.1"
+  generalImplementation "com.google.android.gms:play-services-vision:${safeExtGet('google-services', '17.0.2')}"
+  implementation "com.android.support:exifinterface:${safeExtGet('supportLibVersion', '27.1.0')}"
+  implementation "com.android.support:support-annotations:${safeExtGet('supportLibVersion', '27.1.0')}"
+  implementation "com.android.support:support-v4:${safeExtGet('supportLibVersion', '27.1.0')}"
+  mlkitImplementation "com.google.firebase:firebase-ml-vision:${safeExtGet('firebase-ml-vision', '18.0.2')}"
+  mlkitImplementation "com.google.firebase:firebase-ml-vision-face-model:17.0.2"
 }
 ```
 

diff --git a/android/build.gradle b/android/build.gradle
@@ -26,6 +26,30 @@ android {
     minSdkVersion safeExtGet('minSdkVersion', 16)
     targetSdkVersion safeExtGet('targetSdkVersion', 26)
   }
+
+  flavorDimensions "react-native-camera"
+
+  productFlavors {
+    general {
+      dimension "react-native-camera"
+    }
+    mlkit {
+      dimension "react-native-camera"
+    }
+  }
+
+  sourceSets {
+    main {
+      java.srcDirs = ['src/main/java']
+    }
+    general {
+      java.srcDirs = ['src/general/java']
+    }
+    mlkit {
+      java.srcDirs = ['src/mlkit/java']
+    }
+  }
+
   lintOptions {
     abortOnError false
     warning 'InvalidPackage'
@@ -46,14 +70,16 @@ repositories {
 }
 
 dependencies {
-  def googlePlayServicesVisionVersion = safeExtGet('googlePlayServicesVisionVersion', safeExtGet('googlePlayServicesVersion', '15.0.2'))
+  def googlePlayServicesVisionVersion = safeExtGet('googlePlayServicesVisionVersion', safeExtGet('googlePlayServicesVersion', '17.0.2'))
 
   compileOnly 'com.facebook.react:react-native:+'
   compileOnly 'com.facebook.infer.annotation:infer-annotation:+'
   implementation "com.google.zxing:core:3.3.0"
   implementation "com.drewnoakes:metadata-extractor:2.9.1"
-  implementation "com.google.android.gms:play-services-vision:$googlePlayServicesVisionVersion"
+  generalImplementation "com.google.android.gms:play-services-vision:$googlePlayServicesVisionVersion"
   implementation "com.android.support:exifinterface:${safeExtGet('supportLibVersion', '27.1.0')}"
   implementation "com.android.support:support-annotations:${safeExtGet('supportLibVersion', '27.1.0')}"
   implementation "com.android.support:support-v4:${safeExtGet('supportLibVersion', '27.1.0')}"
+  mlkitImplementation "com.google.firebase:firebase-ml-vision:${safeExtGet('firebase-ml-vision', '18.0.2')}"
+  mlkitImplementation "com.google.firebase:firebase-ml-vision-face-model:${safeExtGet('firebase-ml-vision-face-model', '17.0.2')}"
 }
diff --git a/...e/barcodedetector/BarcodeFormatUtils.java → ...e/barcodedetector/BarcodeFormatUtils.java b/...e/barcodedetector/BarcodeFormatUtils.java → ...e/barcodedetector/BarcodeFormatUtils.java
diff --git a/...ve/barcodedetector/RNBarcodeDetector.java → ...ve/barcodedetector/RNBarcodeDetector.java b/...ve/barcodedetector/RNBarcodeDetector.java → ...ve/barcodedetector/RNBarcodeDetector.java