Skip to content

Commit ff46b1d

Browse files
pgladkowspgladkows
authored
universal_and_benchmark_documentation_updates (#2433)
* universal_and_benchmark_documentation_updates * no proxy update * update benchmark proxy * add version to ubuntu tag * revert ubuntu changes * added localhost * review
1 parent 6a7b271 commit ff46b1d

File tree

2 files changed

+148
-149
lines changed

2 files changed

+148
-149
lines changed

‎demos/benchmark/python/README.md

+148-146
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,8 @@ To check available options use `-h`, `--help` switches:
6868
```bash
6969
docker run benchmark_client --help
7070

71+
Client 2.7
72+
NO_PROXY=localhost no_proxy=localhost python3 /ovms_benchmark_client/main.py --help
7173
usage: main.py [-h] [-i ID] [-c CONCURRENCY] [-a SERVER_ADDRESS]
7274
[-p GRPC_PORT] [-r REST_PORT] [-l] [-b [BS ...]]
7375
[-s [SHAPE ...]] [-d [DATA ...]] [-j] [-m MODEL_NAME]
@@ -77,7 +79,7 @@ usage: main.py [-h] [-i ID] [-c CONCURRENCY] [-a SERVER_ADDRESS]
7779
[--max_value MAX_VALUE] [--min_value MIN_VALUE] [--xrand XRAND]
7880
[--dump_png] [--step_timeout STEP_TIMEOUT]
7981
[--metadata_timeout METADATA_TIMEOUT] [-Y DB_ENDPOINT]
80-
[-y [DB_METADATA ...]] [--print_all] [-ps PRINT_SUMMARY] [--print_time]
82+
[-y [DB_METADATA ...]] [--print_all] [-ps] [--print_time]
8183
[--report_warmup] [--certs_dir CERTS_DIR] [-q STATEFUL_LENGTH]
8284
[--stateful_id STATEFUL_ID] [--stateful_hop STATEFUL_HOP]
8385
[--sync_interval SYNC_INTERVAL]
@@ -93,6 +95,8 @@ The version can be checked by using `--internal_version` switch as follows:
9395
```bash
9496
docker run benchmark_client --internal_version
9597

98+
Client 2.7
99+
NO_PROXY=localhost no_proxy=localhost python3 /ovms_benchmark_client/main.py --internal_version
96100
2.7
97101
```
98102
@@ -105,11 +109,11 @@ docker run --network host benchmark_client -a localhost -r 8000 --list_models
105109

106110
Client 2.7
107111
NO_PROXY=localhost no_proxy=localhost python3 /ovms_benchmark_client/main.py -a localhost -r 8000 --list_models
108-
XI worker: try to send request to endpoint: http://localhost:8000/v1/config
109-
XI worker: received status code is 200.
110-
XI worker: found models and their status:
111-
XI worker: model: resnet50-binary-0001, version: 1 - AVAILABLE
112-
112+
XW worker: Finished execution. If you want to run inference remove --list_models.
113+
XI worker: try to send request to endpoint: http://localhost:8000/v1/config
114+
XI worker: received status code is 200.
115+
XI worker: found models and their status:
116+
XI worker: model: resnet50-binary-0001, version: 1 - AVAILABLE
113117
```
114118
## Sample benchmarks
115119
@@ -123,25 +127,24 @@ docker run --network host benchmark_client -a localhost -r 8000 -l -m resnet50-b
123127

124128
Client 2.7
125129
NO_PROXY=localhost no_proxy=localhost python3 /ovms_benchmark_client/main.py -a localhost -r 8000 -l -m resnet50-binary-0001 -p 9000 -i id
126-
XW id: Finished execution. If you want to run inference remove --list_models.
127-
XI id: try to send request to endpoint: http://localhost:8000/v1/config
128-
XI id: received status code is 200.
129-
XI id: found models and their status:
130-
XI id: model: resnet50-binary-0001, version: 1 - AVAILABLE
131-
XI id: request for metadata of model resnet50-binary-0001...
132-
XI id: Metadata for model resnet50-binary-0001 is downloaded...
133-
XI id: set version of model resnet50-binary-0001: 1
134-
XI id: inputs:
135-
XI id: 0:
136-
XI id: name: 0
137-
XI id: dtype: DT_FLOAT
138-
XI id: tensorShape: {'dim': [{'size': '1'}, {'size': '3'}, {'size': '224'}, {'size': '224'}]}
139-
XI id: outputs:
140-
XI id: 1463:
141-
XI id: name: 1463
142-
XI id: dtype: DT_FLOAT
143-
XI id: tensorShape: {'dim': [{'size': '1'}, {'size': '1000'}]}
144-
130+
XW id: Finished execution. If you want to run inference remove --list_models.
131+
XI id: try to send request to endpoint: http://localhost:8000/v1/config
132+
XI id: received status code is 200.
133+
XI id: found models and their status:
134+
XI id: model: resnet50-binary-0001, version: 1 - AVAILABLE
135+
XI id: request for metadata of model resnet50-binary-0001...
136+
XI id: Metadata for model resnet50-binary-0001 is downloaded...
137+
XI id: set version of model resnet50-binary-0001: 1
138+
XI id: inputs:
139+
XI id: 0:
140+
XI id: name: 0
141+
XI id: dtype: DT_FLOAT
142+
XI id: tensorShape: {'dim': [{'size': '1'}, {'size': '3'}, {'size': '224'}, {'size': '224'}]}
143+
XI id: outputs:
144+
XI id: 1463:
145+
XI id: name: 1463
146+
XI id: dtype: DT_FLOAT
147+
XI id: tensorShape: {'dim': [{'size': '1'}, {'size': '1000'}]}
145148
```
146149
Be sure the model name specified is identical to the model name shown when using
147150
the `--list_models` parameter. A model version is not required but it can be added
@@ -159,127 +162,126 @@ docker run --network host benchmark_client -a localhost -r 8000 -m resnet50-bina
159162
160163
Client 2.7
161164
NO_PROXY=localhost no_proxy=localhost python3 /ovms_benchmark_client/main.py -a localhost -r 8000 -m resnet50-binary-0001 -p 9000 -n 8 --report_warmup --print_all
162-
XI worker: request for metadata of model resnet50-binary-0001...
163-
XI worker: Metadata for model resnet50-binary-0001 is downloaded...
164-
XI worker: set version of model resnet50-binary-0001: 1
165-
XI worker: inputs:
166-
XI worker: 0:
167-
XI worker: name: 0
168-
XI worker: dtype: DT_FLOAT
169-
XI worker: tensorShape: {'dim': [{'size': '1'}, {'size': '3'}, {'size': '224'}, {'size': '224'}]}
170-
XI worker: outputs:
171-
XI worker: 1463:
172-
XI worker: name: 1463
173-
XI worker: dtype: DT_FLOAT
174-
XI worker: tensorShape: {'dim': [{'size': '1'}, {'size': '1000'}]}
175-
XI worker: new random range: 0.0, 255.0
176-
XI worker: batchsize sequence: [1]
177-
XI worker: dataset length (0): 1
178-
XI worker: --> dim: 1
179-
XI worker: --> dim: 3
180-
XI worker: --> dim: 224
181-
XI worker: --> dim: 224
182-
XI worker: Generated data shape: (1, 3, 224, 224)
183-
XI worker: start workload...
184-
XI worker: stop warmup: 374943.047975389
185-
XI worker: stop window: inf
186-
XI worker: Workload started!
187-
XI worker: Warmup normally stopped: 374943.075028319
188-
XI worker: Window normally start: 374943.07504102
189-
XI worker: Window stopped: 374943.354446821
190-
XI worker: total_duration: 0.3065074360347353
191-
XI worker: total_batches: 8
192-
XI worker: total_frames: 8
193-
XI worker: start_timestamp: 374943.047975189
194-
XI worker: stop_timestamp: 374943.354482625
195-
XI worker: pass_batches: 8
196-
XI worker: fail_batches: 0
197-
XI worker: pass_frames: 8
198-
XI worker: fail_frames: 0
199-
XI worker: first_latency: 0.027021727000828832
200-
XI worker: pass_max_latency: 0.0449919409584254
201-
XI worker: fail_max_latency: 0.0
202-
XI worker: brutto_batch_rate: 26.100508697261724
203-
XI worker: brutto_frame_rate: 26.100508697261724
204-
XI worker: netto_batch_rate: 26.127558971388062
205-
XI worker: netto_frame_rate: 26.127558971388062
206-
XI worker: frame_passrate: 1.0
207-
XI worker: batch_passrate: 1.0
208-
XI worker: mean_latency: 0.0382737630061456
209-
XI worker: mean_latency2: 0.0015027467953827884
210-
XI worker: stdev_latency: 0.006153524252994284
211-
XI worker: cv_latency: 0.16077656780197483
212-
XI worker: pass_mean_latency: 0.0382737630061456
213-
XI worker: pass_mean_latency2: 0.0015027467953827884
214-
XI worker: pass_stdev_latency: 0.006153524252994284
215-
XI worker: pass_cv_latency: 0.16077656780197483
216-
XI worker: fail_mean_latency: 0.0
217-
XI worker: fail_mean_latency2: 0.0
218-
XI worker: fail_stdev_latency: 0.0
219-
XI worker: fail_cv_latency: 0.0
220-
XI worker: window_total_duration: 0.27940580097492784
221-
XI worker: window_total_batches: 8
222-
XI worker: window_total_frames: 8
223-
XI worker: window_start_timestamp: 374943.07504102
224-
XI worker: window_stop_timestamp: 374943.354446821
225-
XI worker: window_pass_batches: 8
226-
XI worker: window_fail_batches: 0
227-
XI worker: window_pass_frames: 8
228-
XI worker: window_fail_frames: 0
229-
XI worker: window_first_latency: 0.027021727000828832
230-
XI worker: window_pass_max_latency: 0.0449919409584254
231-
XI worker: window_fail_max_latency: 0.0
232-
XI worker: window_brutto_batch_rate: 28.632190069374655
233-
XI worker: window_brutto_frame_rate: 28.632190069374655
234-
XI worker: window_netto_batch_rate: 26.127558971388062
235-
XI worker: window_netto_frame_rate: 26.127558971388062
236-
XI worker: window_frame_passrate: 1.0
237-
XI worker: window_batch_passrate: 1.0
238-
XI worker: window_mean_latency: 0.0382737630061456
239-
XI worker: window_mean_latency2: 0.0015027467953827884
240-
XI worker: window_stdev_latency: 0.006153524252994284
241-
XI worker: window_cv_latency: 0.16077656780197483
242-
XI worker: window_pass_mean_latency: 0.0382737630061456
243-
XI worker: window_pass_mean_latency2: 0.0015027467953827884
244-
XI worker: window_pass_stdev_latency: 0.006153524252994284
245-
XI worker: window_pass_cv_latency: 0.16077656780197483
246-
XI worker: window_fail_mean_latency: 0.0
247-
XI worker: window_fail_mean_latency2: 0.0
248-
XI worker: window_fail_stdev_latency: 0.0
249-
XI worker: window_fail_cv_latency: 0.0
250-
XI worker: window_hist_latency_4: 2
251-
XI worker: window_hist_latency_9: 1
252-
XI worker: window_hist_latency_8: 5
253-
XI worker: warmup_total_duration: 0.02705443004379049
254-
XI worker: warmup_total_batches: 0
255-
XI worker: warmup_total_frames: 0
256-
XI worker: warmup_start_timestamp: 374943.047973889
257-
XI worker: warmup_stop_timestamp: 374943.075028319
258-
XI worker: warmup_pass_batches: 0
259-
XI worker: warmup_fail_batches: 0
260-
XI worker: warmup_pass_frames: 0
261-
XI worker: warmup_fail_frames: 0
262-
XI worker: warmup_first_latency: inf
263-
XI worker: warmup_pass_max_latency: 0.0
264-
XI worker: warmup_fail_max_latency: 0.0
265-
XI worker: warmup_brutto_batch_rate: 0.0
266-
XI worker: warmup_brutto_frame_rate: 0.0
267-
XI worker: warmup_netto_batch_rate: 0.0
268-
XI worker: warmup_netto_frame_rate: 0.0
269-
XI worker: warmup_frame_passrate: 0.0
270-
XI worker: warmup_batch_passrate: 0.0
271-
XI worker: warmup_mean_latency: 0.0
272-
XI worker: warmup_mean_latency2: 0.0
273-
XI worker: warmup_stdev_latency: 0.0
274-
XI worker: warmup_cv_latency: 0.0
275-
XI worker: warmup_pass_mean_latency: 0.0
276-
XI worker: warmup_pass_mean_latency2: 0.0
277-
XI worker: warmup_pass_stdev_latency: 0.0
278-
XI worker: warmup_pass_cv_latency: 0.0
279-
XI worker: warmup_fail_mean_latency: 0.0
280-
XI worker: warmup_fail_mean_latency2: 0.0
281-
XI worker: warmup_fail_stdev_latency: 0.0
282-
XI worker: warmup_fail_cv_latency: 0.0
165+
XI worker: request for metadata of model resnet50-binary-0001...
166+
XI worker: Metadata for model resnet50-binary-0001 is downloaded...
167+
XI worker: set version of model resnet50-binary-0001: 1
168+
XI worker: inputs:
169+
XI worker: 0:
170+
XI worker: name: 0
171+
XI worker: dtype: DT_FLOAT
172+
XI worker: tensorShape: {'dim': [{'size': '1'}, {'size': '3'}, {'size': '224'}, {'size': '224'}]}
173+
XI worker: outputs:
174+
XI worker: 1463:
175+
XI worker: name: 1463
176+
XI worker: dtype: DT_FLOAT
177+
XI worker: tensorShape: {'dim': [{'size': '1'}, {'size': '1000'}]}
178+
XI worker: new random range: 0.0, 255.0
179+
XI worker: batchsize sequence: [1]
180+
XI worker: dataset length (0): 1
181+
XI worker: --> dim: 1
182+
XI worker: --> dim: 3
183+
XI worker: --> dim: 224
184+
XI worker: --> dim: 224
185+
XI worker: Generated data shape: (1, 3, 224, 224)
186+
XI worker: start workload...
187+
XI worker: stop warmup: 9408188.83686497
188+
XI worker: stop window: inf
189+
XI worker: Workload started!
190+
XI worker: Warmup normally stopped: 9408188.848778868
191+
XI worker: Window normally start: 9408188.848811286
192+
XI worker: Window stopped: 9408188.893217305
193+
XI worker: total_duration: 0.0563836432993412
194+
XI worker: total_batches: 8
195+
XI worker: total_frames: 8
196+
XI worker: start_timestamp: 9408188.836864596
197+
XI worker: stop_timestamp: 9408188.89324824
198+
XI worker: pass_batches: 8
199+
XI worker: fail_batches: 0
200+
XI worker: pass_frames: 8
201+
XI worker: fail_frames: 0
202+
XI worker: first_latency: 0.011858431622385979
203+
XI worker: pass_max_latency: 0.011858431622385979
204+
XI worker: fail_max_latency: 0.0
205+
XI worker: brutto_batch_rate: 141.88512007867135
206+
XI worker: brutto_frame_rate: 141.88512007867135
207+
XI worker: netto_batch_rate: 142.7839056346449
208+
XI worker: netto_frame_rate: 142.7839056346449
209+
XI worker: frame_passrate: 1.0
210+
XI worker: batch_passrate: 1.0
211+
XI worker: mean_latency: 0.00700359046459198
212+
XI worker: mean_latency2: 5.376289226632219e-05
213+
XI worker: stdev_latency: 0.002170855331568294
214+
XI worker: cv_latency: 0.309963202809113
215+
XI worker: pass_mean_latency: 0.00700359046459198
216+
XI worker: pass_mean_latency2: 5.376289226632219e-05
217+
XI worker: pass_stdev_latency: 0.002170855331568294
218+
XI worker: pass_cv_latency: 0.309963202809113
219+
XI worker: fail_mean_latency: 0.0
220+
XI worker: fail_mean_latency2: 0.0
221+
XI worker: fail_stdev_latency: 0.0
222+
XI worker: fail_cv_latency: 0.0
223+
XI worker: window_total_duration: 0.044406019151210785
224+
XI worker: window_total_batches: 8
225+
XI worker: window_total_frames: 8
226+
XI worker: window_start_timestamp: 9408188.848811286
227+
XI worker: window_stop_timestamp: 9408188.893217305
228+
XI worker: window_pass_batches: 8
229+
XI worker: window_fail_batches: 0
230+
XI worker: window_pass_frames: 8
231+
XI worker: window_fail_frames: 0
232+
XI worker: window_first_latency: 0.011858431622385979
233+
XI worker: window_pass_max_latency: 0.011858431622385979
234+
XI worker: window_fail_max_latency: 0.0
235+
XI worker: window_brutto_batch_rate: 180.15575710037206
236+
XI worker: window_brutto_frame_rate: 180.15575710037206
237+
XI worker: window_netto_batch_rate: 142.7839056346449
238+
XI worker: window_netto_frame_rate: 142.7839056346449
239+
XI worker: window_frame_passrate: 1.0
240+
XI worker: window_batch_passrate: 1.0
241+
XI worker: window_mean_latency: 0.00700359046459198
242+
XI worker: window_mean_latency2: 5.376289226632219e-05
243+
XI worker: window_stdev_latency: 0.002170855331568294
244+
XI worker: window_cv_latency: 0.309963202809113
245+
XI worker: window_pass_mean_latency: 0.00700359046459198
246+
XI worker: window_pass_mean_latency2: 5.376289226632219e-05
247+
XI worker: window_pass_stdev_latency: 0.002170855331568294
248+
XI worker: window_pass_cv_latency: 0.309963202809113
249+
XI worker: window_fail_mean_latency: 0.0
250+
XI worker: window_fail_mean_latency2: 0.0
251+
XI worker: window_fail_stdev_latency: 0.0
252+
XI worker: window_fail_cv_latency: 0.0
253+
XI worker: window_hist_latency_1: 1
254+
XI worker: window_hist_latency_0: 7
255+
XI worker: warmup_total_duration: 0.011916300281882286
256+
XI worker: warmup_total_batches: 0
257+
XI worker: warmup_total_frames: 0
258+
XI worker: warmup_start_timestamp: 9408188.836862568
259+
XI worker: warmup_stop_timestamp: 9408188.848778868
260+
XI worker: warmup_pass_batches: 0
261+
XI worker: warmup_fail_batches: 0
262+
XI worker: warmup_pass_frames: 0
263+
XI worker: warmup_fail_frames: 0
264+
XI worker: warmup_first_latency: inf
265+
XI worker: warmup_pass_max_latency: 0.0
266+
XI worker: warmup_fail_max_latency: 0.0
267+
XI worker: warmup_brutto_batch_rate: 0.0
268+
XI worker: warmup_brutto_frame_rate: 0.0
269+
XI worker: warmup_netto_batch_rate: 0.0
270+
XI worker: warmup_netto_frame_rate: 0.0
271+
XI worker: warmup_frame_passrate: 0.0
272+
XI worker: warmup_batch_passrate: 0.0
273+
XI worker: warmup_mean_latency: 0.0
274+
XI worker: warmup_mean_latency2: 0.0
275+
XI worker: warmup_stdev_latency: 0.0
276+
XI worker: warmup_cv_latency: 0.0
277+
XI worker: warmup_pass_mean_latency: 0.0
278+
XI worker: warmup_pass_mean_latency2: 0.0
279+
XI worker: warmup_pass_stdev_latency: 0.0
280+
XI worker: warmup_pass_cv_latency: 0.0
281+
XI worker: warmup_fail_mean_latency: 0.0
282+
XI worker: warmup_fail_mean_latency2: 0.0
283+
XI worker: warmup_fail_stdev_latency: 0.0
284+
XI worker: warmup_fail_cv_latency: 0.0
283285
```
284286
285287
## Dynamic models benchmarking

‎demos/universal-sentence-encoder/README.md

-3
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,6 @@ tree universal-sentence-encoder-multilingual/
1515

1616
universal-sentence-encoder-multilingual/
1717
└── 1
18-
├── assets
1918
├── saved_model.pb
2019
└── variables
2120
├── variables.data-00000-of-00001
@@ -89,7 +88,6 @@ Start TFS container:
8988
docker run -it -p 8500:8500 -p 9500:9500 -v $(pwd)/universal-sentence-encoder-multilingual:/models/usem -e MODEL_NAME=usem tensorflow/serving --port=9500 --rest_api_port=8500
9089
```
9190

92-
9391
Run the client
9492
```bash
9593
python model_server/demos/universal-sentence-encoder/send_strings.py --grpc_port 9500 --input_name inputs --output_name outputs --string "I enjoy taking long walks along the beach with my dog."
@@ -104,4 +102,3 @@ Output subset [-0.00552387 0.00599531 -0.0148055 0.01098951 -0.09355522 -0.08
104102
```
105103

106104
> NOTE: Do not use this model with `--cache_dir`, the model does not support caching.
107-

0 commit comments

Comments
 (0)