Skip to content
This repository was archived by the owner on Jul 22, 2024. It is now read-only.

Commit 3981478

Browse files
csbujaj-windsor
csbuja
authored and
j-windsor
committedJun 29, 2019
Csbuja added ml backend deployment json (#9)
* initial deployment automation * next, delete batch from the template * Removed Batch pools because we cannot reference the images in our subscription in the template * typos and switched jobs to tasks * pr updates * json * update * typo
1 parent 7edbbbe commit 3981478

File tree

2 files changed

+214
-4
lines changed

2 files changed

+214
-4
lines changed
 

‎README.md

+34-4
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ There are currently no tests, but we would love it if someone would contribute s
6565

6666
The service is an ASP.NET Core application that has many Azure dependencies. We will first get these dependencies set up.
6767

68-
- Visit the Azure Portal and choose to create a new resource of type "Template Deployment".
68+
- Visit the Azure Portal and create a new resource of type "Template Deployment".
6969
On the next page, select "Build your own template in the editor", and upload the template file `/deployment/az-service-template.json`.
7070
On the next page, fill in the resource and resource group names. Purchase this resource group.
7171
- Create an AAD app for the service. For more information on how to create an AAD app,
@@ -78,18 +78,48 @@ visit the [azure docs](https://docs.microsoft.com/en-us/azure/active-directory/d
7878
you can populate the rest of the secrets.
7979
- On the newly created Cosmos Document DB account, create two new containers named "BatchJob" and "Document".
8080
- Download and install [Visual Studio 2019](https://visualstudio.microsoft.com/downloads) with the "ASP.NET and web development" workload.
81-
- In `/BrowseCloud.Service/BrowseCloud.Service/appsettings.json`, Configure your development environment using the information from the services you just created.
81+
- In `/BrowseCloud.Service/BrowseCloud.Service/appsettings.json`, configure your development environment using the information from the services you just created.
8282
- You can then build and run using Visual Studio's built in build and run feature.
8383

8484
This can be built and deployed to the Azure App Service generated in the steps above for everyday use.
8585
The easiest method is to right click on the BrowseCloud.Service project and "Publish", but we should recommend a CI/CD pipeline of some type.
8686
We have our Azure DevOps build pipelines checked in as yaml files which you are welcomed to use.
8787

88-
There are currently no tests, but we would love it if someone would contribute some 😉
88+
There are currently no tests on the Service, but we welcome contribution on this front.
8989

9090
## Trainer Jobs
91+
This is the machine learning backend that powers BrowseCloud. It has many Azure dependencies.
92+
93+
- Visit the Azure Portal and create a new resource of type "Template Deployment".
94+
On the next page, select "Build your own template in the editor", and upload the template file `/deployment/az-ml-backend-template.json`.
95+
On the next page, fill in the resource and resource group names. Purchase this resource group.
96+
97+
Next, we will setup our VM. The work to setup dependencies on a machine in the cloud like this is automatable, but it hasn't been done.
98+
- Visit the Azure Portal and choose to create a new resource of type "Windows Server 2016 Datacenter". In this initial setup, make sure you have RDP enabled to setup the VM.
99+
- RDP into the non-production VM and [follow the setup instructions to get the CountingGridsPy library running on the VM](https://github.com/microsoft/browsecloud/wiki/Environment-Setup-&-Dependencies-to-run-CountingGridsPy-Locally). In your production instance of the VM, we recommend that you have RDP turned off.
100+
- Save your VM as an image within the new virtual machine resource on the Azure Portal. This will destabilize the VM, so you should delete the VM.
101+
102+
- Next, we'll take a look at the Batch resource you generated from the template. The purpose of Batch is to manage and scale computational power with the machine learning work to do.
103+
104+
Create two jobs and two pools within this Batch resource, one for your dev environment and another for your production environment. You can do this by using the Azure portal or by using `\Batch\Batch\src\deployBrowseCloudBatchPool.py`. In our design, jobs are permenant, and each training request is a task underneath each job.
105+
106+
We recommend that you scale the number of VMs elastically with the number of tasks running on your queue, so work can be done in parallel. You can even have multiple tasks running on the same machine using Batch. Lastly, recommend that you always have one Windows VM running and ready to go due to in the autoScale Formula.
107+
108+
An example scaling configuration could be:
109+
110+
```json
111+
"scaleSettings": {
112+
"autoScale": {
113+
"formula": "maxNumberofVMs = 5;sample =$PendingTasks.GetSample(10);pendingTaskSamplePercent = avg(sample);startingNumberOfVMs = 1; pendingTaskSamples = pendingTaskSamplePercent < 2 ? startingNumberOfVMs : avg($PendingTasks.GetSample(180 * TimeInterval_Second));$TargetDedicatedNodes=min(maxNumberofVMs, pendingTaskSamples);",
114+
"evaluationInterval": "PT5M"
115+
}
116+
}
117+
```
118+
119+
We also recommend that you use a more powerful VM in your production instance than in your development instance. We use "vmSize" of "STANDARD_D16_V3" on our production site for training new models. We use a "vmSize" of "STANDARD_A1" in our development instance.
120+
121+
- In `/Batch/Batch/src/metadata.json` and `/Batch/Batch/src/keys.json` (which are not checked into this repo), configure your development environment using the information from the services you just created.
91122

92-
TODO
93123

94124
# Contributing
95125
This project welcomes contributions and suggestions. Most contributions require you to

‎deployment/az-ml-backend-template.json

+180
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,180 @@
1+
{
2+
"$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
3+
"contentVersion": "1.0.0.0",
4+
"parameters": {
5+
"batchAccounts_trainer_name": {
6+
"type": "string"
7+
},
8+
"storageAccounts_browseclouddiag_name": {
9+
"type": "string"
10+
},
11+
"storageAccounts_browsecloudapppackage_name": {
12+
"type": "string"
13+
},
14+
"storageAccounts_browsecloudmodelfiles_name": {
15+
"type": "string"
16+
},
17+
"storageAccounts_browsecloudtrainingdata_name": {
18+
"type": "string"
19+
},
20+
"accounts_browsecloud_sentiment_name": {
21+
"type": "string"
22+
}
23+
},
24+
"variables": {},
25+
"resources": [
26+
{
27+
"type": "Microsoft.CognitiveServices/accounts",
28+
"apiVersion": "2016-02-01-preview",
29+
"name": "[parameters('accounts_browsecloud_sentiment_name')]",
30+
"location": "westus2",
31+
"sku": {
32+
"name": "S"
33+
},
34+
"kind": "TextAnalytics",
35+
"properties": {}
36+
},
37+
{
38+
"type": "Microsoft.Storage/storageAccounts",
39+
"apiVersion": "2019-04-01",
40+
"name": "[parameters('storageAccounts_browsecloudapppackage_name')]",
41+
"location": "eastus",
42+
"sku": {
43+
"name": "Standard_RAGRS",
44+
"tier": "Standard"
45+
},
46+
"kind": "StorageV2",
47+
"properties": {
48+
"networkAcls": {
49+
"bypass": "AzureServices",
50+
"virtualNetworkRules": [],
51+
"ipRules": [],
52+
"defaultAction": "Allow"
53+
},
54+
"supportsHttpsTrafficOnly": true,
55+
"encryption": {
56+
"services": {
57+
"file": {
58+
"enabled": true
59+
},
60+
"blob": {
61+
"enabled": true
62+
}
63+
},
64+
"keySource": "Microsoft.Storage"
65+
},
66+
"accessTier": "Hot"
67+
}
68+
},
69+
{
70+
"type": "Microsoft.Storage/storageAccounts",
71+
"apiVersion": "2019-04-01",
72+
"name": "[parameters('storageAccounts_browseclouddiag_name')]",
73+
"location": "eastus",
74+
"sku": {
75+
"name": "Standard_LRS",
76+
"tier": "Standard"
77+
},
78+
"kind": "Storage",
79+
"properties": {
80+
"networkAcls": {
81+
"bypass": "AzureServices",
82+
"virtualNetworkRules": [],
83+
"ipRules": [],
84+
"defaultAction": "Allow"
85+
},
86+
"supportsHttpsTrafficOnly": false,
87+
"encryption": {
88+
"services": {
89+
"file": {
90+
"enabled": true
91+
},
92+
"blob": {
93+
"enabled": true
94+
}
95+
},
96+
"keySource": "Microsoft.Storage"
97+
}
98+
}
99+
},
100+
{
101+
"type": "Microsoft.Storage/storageAccounts",
102+
"apiVersion": "2019-04-01",
103+
"name": "[parameters('storageAccounts_browsecloudmodelfiles_name')]",
104+
"location": "eastus",
105+
"sku": {
106+
"name": "Standard_RAGRS",
107+
"tier": "Standard"
108+
},
109+
"kind": "StorageV2",
110+
"properties": {
111+
"networkAcls": {
112+
"bypass": "AzureServices",
113+
"virtualNetworkRules": [],
114+
"ipRules": [],
115+
"defaultAction": "Allow"
116+
},
117+
"supportsHttpsTrafficOnly": true,
118+
"encryption": {
119+
"services": {
120+
"file": {
121+
"enabled": true
122+
},
123+
"blob": {
124+
"enabled": true
125+
}
126+
},
127+
"keySource": "Microsoft.Storage"
128+
},
129+
"accessTier": "Hot"
130+
}
131+
},
132+
{
133+
"type": "Microsoft.Storage/storageAccounts",
134+
"apiVersion": "2019-04-01",
135+
"name": "[parameters('storageAccounts_browsecloudtrainingdata_name')]",
136+
"location": "centralus",
137+
"sku": {
138+
"name": "Standard_RAGRS",
139+
"tier": "Standard"
140+
},
141+
"kind": "StorageV2",
142+
"properties": {
143+
"networkAcls": {
144+
"bypass": "AzureServices",
145+
"virtualNetworkRules": [],
146+
"ipRules": [],
147+
"defaultAction": "Allow"
148+
},
149+
"supportsHttpsTrafficOnly": true,
150+
"encryption": {
151+
"services": {
152+
"file": {
153+
"enabled": true
154+
},
155+
"blob": {
156+
"enabled": true
157+
}
158+
},
159+
"keySource": "Microsoft.Storage"
160+
},
161+
"accessTier": "Hot"
162+
}
163+
},
164+
{
165+
"type": "Microsoft.Batch/batchAccounts",
166+
"apiVersion": "2017-09-01",
167+
"name": "[parameters('batchAccounts_trainer_name')]",
168+
"location": "eastus",
169+
"dependsOn": [
170+
"[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccounts_browsecloudapppackage_name'))]"
171+
],
172+
"properties": {
173+
"autoStorage": {
174+
"storageAccountId": "[resourceId('Microsoft.Storage/storageAccounts', parameters('storageAccounts_browsecloudapppackage_name'))]"
175+
},
176+
"poolAllocationMode": "BatchService"
177+
}
178+
}
179+
]
180+
}

0 commit comments

Comments
 (0)