Update telemetry docs #29

ronaktruss · 2021-10-21T20:54:13Z

I have added a link to some instructions on how to view logs in the load testing environment.

rogeruiz · 2021-10-26T14:32:38Z

@ronaktruss I haven't been able to get the Docker Compose commands from these docs to work on transcom/mymove#7541. Do they work for you locally?

ronaktruss · 2021-10-26T14:49:43Z

@ronaktruss I haven't been able to get the Docker Compose commands from these docs to work on transcom/mymove#7541. Do they work for you locally?

@rogeruiz I ran into some issues as well, but was able to resolve them by increasing the memory resources available to docker (it was at 2gb and I increased it to 8 or 12). We can pair to see if this resolves the issue and I can also add that those instructions to the docs.

rogeruiz · 2021-10-26T15:54:36Z

After pairing on this briefly with @ronaktruss, we've found out a couple of things. I'm going to use this space to discuss the Kibana stuff that we found.

After bumping up my memory limits, I was able to get things working locally. Once that was done. I saw that there are a couple of Services and Transactions that seem to be weird to me locally. It'll most likely go away in the future as the feature progresses, but it's good to document this for folks working on the feature and building this out.

Services are all named unknown_service_milmove_gin and all local Transactions besides logging in and logging out are called server-no-tls. I'm not sure where this is coming from, but it seems like it's a fix with big implications for the usefulness of this. The login/logout functionality seems to show up fine in the Kibana dashboard, so I'd probably look there first to see why those are unique to all the other routes.

📸 Some screenshots of what this looks like

Working Kibana Dashboard with Services and Transactions

Captura de Pantalla 2021-10-26 a la(s) 11 52 31

Hypothesis that a local URL shows up as `server-no-tls` in Kibana

Captura de Pantalla 2021-10-26 a la(s) 11 52 40

…onaktruss-patch-1

YanZ777

Thanks for updating the docs. Just some minor comments here and there.

docs/tools/telemetry/running-telemetry-locally.md

YanZ777 · 2021-10-27T04:51:50Z

docs/tools/telemetry/running-telemetry-locally.md

+
+  ![Kibana home page](../../../static/img/telemetry/kibana-home-page.png)
+
+1. To start collecting information you must run the mymove app locally. To do this navigate to the mymove repo and run the command `make client_run` to start up the local server. You can then navigate through the app to start collecting data.


Do they also have to do make server_run?

Yeah that was in the previous step with the Open Telemetry environment variables. I think this section is implying that to show data in Kibana you need to navigate through the application.

Ah! Should have read up a bit! Thanks for the clarification.

YanZ777 · 2021-10-27T04:52:27Z

docs/tools/telemetry/running-telemetry-locally.md

+
+  ![Services Dashboard](../../../static/img/telemetry/services-dashboard.png)
+
+  Note: Services are all named unknown_service_milmove_gin and all local transactions besides logging in and logging out are called server-no-tls. This behavior is no ideal and should be looked into. Utilizing local urls such as `officelocal:3000` may be the cause of the lack of semantic naming.


typo:
no -> not

I think we should leave out the "This behavior is not ideal and should be looked into".
Should there be a Jira ticket to fix this sort of thing?

Ooooh I like this idea. Instead of detailing the issue/bug/task in documentation, we can link to a Jira ticket that has the context for the story/bug/chore/etc.

rogeruiz

~~This document needs to be migrated due to #36. I have other suggestions as well. but as-is, this document would not be linked to anything given the changes from #36. Sorry @ronaktruss~~

this request changes was a mistake.

rogeruiz

lgtm 🌈

Thanks for doing this @ronaktruss. I added some suggested changes but I don't want to hold up the ~~review~~ merge of this PR any longer so I'm approving it. But feel free to incorporate these now, otherwise I'll add them in another PR in the future.

rogeruiz · 2021-10-27T16:02:59Z

docs/tools/telemetry/running-telemetry-locally.md

+  You may see an error message like: `ERROR: for apm-server  Container "4aa42b9f5715" is unhealthy` or `ERROR: for apm-server  Container "4aa42b9f5715" is unhealthy` this may be an indication that docker doesn't have enough memory. To resolve this issue open up Docker Desktop. Click the gear icon in the header as shown below:
+
+  ![docker settings](../../../static/img/telemetry/docker_settings.png)
+
+  You can then click `Resources` on the left nav bar and modify the `Memory` slider to `8.00 GB`. Click `Apply & Restart`. Once you have done this rerun the original command.
+
+  ![changed docker settings](../../../static/img/telemetry/docker_resources.png)


I would like use to use Docusaurus Admonitions for big call outs like this.

This is what they look like visually

Here's a helpful GitHub Suggested Change for adding them in a single click to this PR.

Suggested change

You may see an error message like: `ERROR: for apm-server Container "4aa42b9f5715" is unhealthy` or `ERROR: for apm-server Container "4aa42b9f5715" is unhealthy` this may be an indication that docker doesn't have enough memory. To resolve this issue open up Docker Desktop. Click the gear icon in the header as shown below:

![docker settings](../../../static/img/telemetry/docker_settings.png)

You can then click `Resources` on the left nav bar and modify the `Memory` slider to `8.00 GB`. Click `Apply & Restart`. Once you have done this rerun the original command.

![changed docker settings](../../../static/img/telemetry/docker_resources.png)

:::danger You may encounter errors

Docker Compose may report errors similar to the following: `ERROR: for

apm-server Container "4aa42b9f5715" is unhealthy` or `ERROR: for apm-server

Container "4aa42b9f5715" is unhealthy` this may be an indication that docker

doesn't have enough memory. To resolve this issue open up Docker Desktop. Click

the gear icon in the header as shown below:

![docker settings](/img/telemetry/docker_settings.png)

You can then click `Resources` on the left nav bar and modify the `Memory`

slider to `8.00 GB`. Click `Apply & Restart`. Once you have done this rerun the

original command.

![changed docker settings](/img/telemetry/docker_resources.png)

:::

Thanks for this feedback! I didn't know these existed. For now I will be merging these changes as is, but will be adding the admonitions in a separate PR.

rogeruiz · 2021-10-27T16:04:23Z

docs/tools/telemetry/running-telemetry-locally.md


+If you see the following error `context deadline exceeded` this is an indication that it can't reach the telemetry server.


I would like use to use Docusaurus Admonitions for big call outs like this.

This is what they look like visually

Here's a helpful GitHub Suggested Change for adding them in a single click to this PR.

Suggested change

If you see the following error `context deadline exceeded` this is an indication that it can't reach the telemetry server.

:::danger Context deadline exceeded

If you see the following error `context deadline exceeded` this is an indication

that it can't reach the telemetry server.

:::

YanZ777

I have no further comments, thanks for updating this stuff!
The Admonitions that Roger pointed out would be nice, but I am not going to block.

Fix spelling on link

acc1da8

ronaktruss added the A-Team label Oct 21, 2021

ronaktruss added 2 commits October 25, 2021 17:42

Update link formatting to follow standard md rules

92c7746

Add page about viewing telemetry logs

600f2e4

ronaktruss changed the title ~~Fix spelling on link~~ Update telemetry docs Oct 25, 2021

ronaktruss marked this pull request as ready for review October 25, 2021 21:46

ronaktruss requested a review from a team October 25, 2021 21:46

ronaktruss mentioned this pull request Oct 25, 2021

MB-9337 Enable otel sidecar in loadtest env transcom/mymove#7611

Merged

1 task

rogeruiz mentioned this pull request Oct 26, 2021

MB-9609 migrate tools section #36

Merged

ronaktruss added 3 commits October 26, 2021 16:43

Merge branch 'main' of https://github.com/transcom/mymove-docs into r…

c3b9836

…onaktruss-patch-1

Add images and depth to instructions for better clarity

97d1003

Reorganize structure and add troubleshooting section

14c4d8f

ronaktruss requested a review from rogeruiz October 27, 2021 00:08

YanZ777 reviewed Oct 27, 2021

View reviewed changes

Fix typo and move bug to Jira

99f3bd2

rogeruiz previously requested changes Oct 27, 2021

View reviewed changes

rogeruiz approved these changes Oct 27, 2021

View reviewed changes

YanZ777 approved these changes Oct 27, 2021

View reviewed changes

ronaktruss merged commit fde37dd into main Oct 27, 2021

ronaktruss deleted the ronaktruss-patch-1 branch October 27, 2021 19:02

ronaktruss mentioned this pull request Oct 28, 2021

Update links to no longer use relative urls and add admonitions #37

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update telemetry docs #29

Update telemetry docs #29

ronaktruss commented Oct 21, 2021 •

edited

Loading

rogeruiz commented Oct 26, 2021

ronaktruss commented Oct 26, 2021 •

edited

Loading

rogeruiz commented Oct 26, 2021

Working Kibana Dashboard with Services and Transactions

Hypothesis that a local URL shows up as `server-no-tls` in Kibana

YanZ777 left a comment

YanZ777 Oct 27, 2021

rogeruiz Oct 27, 2021

YanZ777 Oct 27, 2021

YanZ777 Oct 27, 2021

YanZ777 Oct 27, 2021

rogeruiz Oct 27, 2021

ronaktruss Oct 27, 2021

rogeruiz left a comment •

edited

Loading

rogeruiz left a comment •

edited

Loading

rogeruiz Oct 27, 2021

ronaktruss Oct 27, 2021

rogeruiz Oct 27, 2021

YanZ777 left a comment


		![Kibana home page](../../../static/img/telemetry/kibana-home-page.png)

		1. To start collecting information you must run the mymove app locally. To do this navigate to the mymove repo and run the command `make client_run` to start up the local server. You can then navigate through the app to start collecting data.


		![Services Dashboard](../../../static/img/telemetry/services-dashboard.png)

		Note: Services are all named unknown_service_milmove_gin and all local transactions besides logging in and logging out are called server-no-tls. This behavior is no ideal and should be looked into. Utilizing local urls such as `officelocal:3000` may be the cause of the lack of semantic naming.


		If you see the following error `context deadline exceeded` this is an indication that it can't reach the telemetry server.

Update telemetry docs #29

Update telemetry docs #29

Conversation

ronaktruss commented Oct 21, 2021 • edited Loading

rogeruiz commented Oct 26, 2021

ronaktruss commented Oct 26, 2021 • edited Loading

rogeruiz commented Oct 26, 2021

Working Kibana Dashboard with Services and Transactions

Hypothesis that a local URL shows up as server-no-tls in Kibana

YanZ777 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rogeruiz left a comment • edited Loading

Choose a reason for hiding this comment

rogeruiz left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

YanZ777 left a comment

Choose a reason for hiding this comment

ronaktruss commented Oct 21, 2021 •

edited

Loading

ronaktruss commented Oct 26, 2021 •

edited

Loading

Hypothesis that a local URL shows up as `server-no-tls` in Kibana

rogeruiz left a comment •

edited

Loading

rogeruiz left a comment •

edited

Loading