Restructure Documents folders, separate design from compliance requir…

…ements, etc. (sonic-net#45)

* Updated diagram

* Update dash-test-workflow-saithrift.md

Added reminder notes.

* Begin outlining compliance specs.

* Fix links

* Add missing diagram. Should jhave already been in main.

* Compliance wip.

* Spec relationship diagram.

* Describe doc org, update diagram.

* docs org diagram, desription

* Note on compliance doc.

* Typos.

* Restructure Documentation per discussion with team, add ToCs etc.

* Move images into respective design/requirement folders. Delete obs dash-running icons.

* Complete ToC links etc.

* Fix ToCs, titles, added load-bal-service.

* Change formatting of Toc backlinks.

* fix filenames, links

* image url error

* Update Documentation/README.md

Co-authored-by: KrisNey-MSFT <[email protected]>

* Update Documentation/README.md

Co-authored-by: KrisNey-MSFT <[email protected]>

* Update Documentation/README.md

Co-authored-by: KrisNey-MSFT <[email protected]>

* Update diagram.

* Added services, placeholder directories/READMEs. Restructured top ToC into two sections for baseline and per-service.

* Add back accidentally-deleted service tunnel private link items.

* Minor updates to README; added context notes to scale testing doc.

* Added helpful liks to GitHub workflows (issues, forking, PRs).

* Rename Assets -> assets, Documentation -> documentation, fix broken image link in SDN doc.

* underlay->overlay in DASH HLD

* Fix label

* Fix links to test HLD.

* fix links

* Clarify where docs are stored

* Update documentation/README.md

Co-authored-by: Michael Miele <[email protected]>

* Update documentation/README.md

Co-authored-by: Michael Miele <[email protected]>

* Update documentation/README.md

Co-authored-by: Michael Miele <[email protected]>

* Update documentation/README.md

Co-authored-by: Michael Miele <[email protected]>

* Update documentation/README.md

Co-authored-by: Michael Miele <[email protected]>

* Moved git rules to top dir, added link.

* Moved contents higher on page.

Co-authored-by: Michael Miele <[email protected]>
Co-authored-by: Chris Sommers <[email protected]>
Co-authored-by: KrisNey-MSFT <[email protected]>
Co-authored-by: Michael Miele <[email protected]>

README.md

@@ -11,19 +11,22 @@ Future innovations for in-service software upgrades and ultra-high availability
 We hope that DASH will have the same success as SONiC for switches and also be widely adopted as a major Open NOS for Programmable Technologies (including SmartNICs) to supercharge a variety of cloud and enterprise applications. 
 
 ## Where to Start?
+Go to the [Documentation table of contents](documentation/README.md) for access to all design and requirements documents.
 
-Please begin with 
-1. The [SDN Packet Transforms](Documentation/sdn-features-packet-transforms.md) document, this facilitates understanding of the program goal and the 7 networking scenarios that Azure has defined.  Next...
-2. Peruse the [dash-high-level-design](Documentation/dash-high-level-design.md) for an overview of the architecture, and 
-3. [Program Scale Testing Requirements - Draft](Documentation/program-scale-testing-requirements-draft.md) for an example of a test to stress DPU/NIC hardware.
+For a quick technical deep-dive, please begin with:
+1. The [SDN Packet Transforms](documentation/general/design/sdn-features-packet-transforms.md) document, this facilitates understanding of the program goal and the 7 networking scenarios that Azure has defined.  Next...
+2. Peruse the [dash-high-level-design](documentation/general/design/dash-high-level-design.md) for an overview of the architecture, and 
+3. [Program Scale Testing Requirements - Draft](documentation/general/requirements/program-scale-testing-requirements-draft.md) for an example of a test to stress DPU/NIC hardware.
 
 The API and Object Model for VNET<->VNET is in draft; the remaining services will be posted over as we move forward.
 
 DASH Testing is covered  under the [test/](test/README.md) directory and is a work in progress.
 
 ## I Want To Contribute!
 
-This project welcomes contributions and suggestions.  We are happy to have the Community involved via submission of **Issues and Pull Requests** (with substantive content or even just fixes). We are hoping for the documents to become a community process with active engagement.  PRs can be reviewed by by any number of people, and a maintainer may accept.
+This project welcomes contributions and suggestions.  We are happy to have the Community involved via submission of **Issues and Pull Requests** (with substantive content or even just fixes). We are hoping for the documents, test framework, etc. to become a community process with active engagement.  PRs can be reviewed by by any number of people, and a maintainer may accept.
+
+See [GitHub Basic Process](doc-github-rules.md).
 
 Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
 the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.
@@ -46,5 +49,5 @@ Any use of third-party trademarks or logos are subject to those third-party's po
 
 <!-- dash icon -->
 <div align="center">
-<img src="Documentation/images/icons/dash-icon-xlarge.png" style="align:center;"/>
+<img src="documentation/images/icons/dash-icon-xlarge.png" style="align:center;"/>
 <div/> 

documentation/README.md

@@ -0,0 +1,68 @@
+[[ << Back to DASH top-level README ](../README.md)]
+# DASH Documentation 
+
+Documentation comprises system descriptions, High-level design (HLD) documents and detailed compliance requirements. These are contained in the [DASH/documentation](./) directory and subdirectories.
+
+The testing framework, methodology, documentation and testing artifacts are stored in the [DASH/test](../test) directory
+
+See also DASH [FAQ](https://github.com/Azure/DASH/wiki/FAQ) and [Glossary](https://github.com/Azure/DASH/wiki/Glossary). 
+
+# Contents
+## Baseline Specifications and Requirements
+All DASH devices shall conform to the following design specifications and compliance requirements:
+
+| Topic   | Links to Folders |
+| ------- | ---------|
+| General Architecture and Requirements| [Parent Folder](general/README.md) \| [ Design](general/design/README.md) \| [Compliance Requirements](general/requirements/README.md)|
+| Dataplane                            | [Parent Folder](dataplane/README.md) \| [ Design](dataplane/design/README.md) \| [Compliance Requirements](dataplane/requirements/README.md)|
+| High-Availability (HA)                                                    | [Parent Folder](high-avail/README.md) \| [ Design](high-avail/design/README.md) \| [Compliance Requirements](high-avail/requirements/README.md)|
+| gNMI Northbound API                  | [Parent Folder](gnmi/README.md) \| [ Design](gnmi/design/README.md) \| [Compliance Requirements](gnmi/requirements/README.md)|
+| SAI Southbound API                   | [Parent Folder](sai/README.md) \| [ Design](sai/design/README.md) \| [Compliance Requirements](sai/requirements/README.md)|
+
+## Service Specifications and Requirements
+DASH devices may implement one or more of the following services.
+They shall conform to each service's design specificaitons and compliance requirements.
+| Topic   | Links to Folders |
+| ------- | ---------|
+| Load Balancer Service                | [Parent Folder](load-bal-service/README.md) \| [ Design](load-bal-service/design/README.md) \| [Compliance Requirements](load-bal-service/requirements/README.md)|
+| VNET-to-VNET Service                         | [Parent Folder](vnet2vnet-service/README.md) \| [ Design](vnet2vnet-service/design/README.md) \| [Compliance Requirements](vnet2vnet-service/requirements/README.md)|
+| Service Tunnel & Private Link Service                          | [Parent Folder](stpl-service/README.md) \| [ Design](stpl-service/design/README.md) \| [Compliance Requirements](stpl-service/requirements/README.md)|
+| VNET Peering Service                          | [Parent Folder](vnet-peering-service/README.md) \| [ Design](vnet-peering-service/design/README.md) \| [Compliance Requirements](vnet-peering-service/requirements/README.md)|
+| Express Route (ER) Service                    |  [Parent Folder](express-route-service/README.md) \| [ Design](express-route-service/design/README.md) \| [Compliance Requirements](express-route-service/requirements/README.md)|
+| Encryption Gateway Service                    |  [Parent Folder](encrypt-gw-service/README.md) \| [ Design](encrypt-gw-service/design/README.md) \| [Compliance Requirements](encrypt-gw-service/requirements/README.md)|
+
+
+# Organization of Design & Requirements Documents
+Documentation consists of separate, but related *System Descriptions* (HLDs, architecture, theory of operations, etc.) and *Compliance Requirements* (hard specifications, typically numerical  but also behavioral). These two types of documents are deliberately kept separated, see [Relationships and Flow of Documents](#relationships-and-flow-of-documents).
+
+Documentation is organized into folders as follows. Each feature or topic has all the high-level specs  and the compliance requirements in the same parent folder, e.g. General, High-Availability, etc., making it easier to access related information about one topic. As the complexity grows, this helps keep things organized according to "functional topic." 
+```
+topic1
+    design
+        topic1 High-level Descriptions and architecture
+    requirements
+        topic1 compliance Requirements
+
+topic2
+    design
+        topic2 High-level Descriptions
+    requirements
+        topic2 compliance Requirements
+
+etc
+...
+```
+
+# Relationships and Flow of Documents
+The diagram below shows how High-Level Descriptions beget Compliance requirements, compliance requirements beget test cases, and test cases are executed by test scripts to produce Test Results.
+
+![dash-specs-flow](images/general/dash-specs-flow.svg).
+
+Some of the guiding principles for this aproach are:
+* Define the objectives and the design or proposal separately from performance and requirement details.
+* Describe **hard requirements** separately from the design and architecture descriptions This allows the requirements to be easily defined, maintained, and referenced from other downstream "consumers," e.g., **test cases**. All requirements must be identified with some designator which allows traceability in test cases, scripts and results.
+* We encourage the creation of simultaneous **human** and **machine-readable** data which can **drive test cases**.  
+* We must avoid burying test parameters into the test scripts. This allows the requirements to be maintained/defined independently from the (often complex) code which executes tests. 
+* Many projects exist where only a programmer can locate and ferret out actual test criteria, often expressed as hard-coded constants buried within thousands of lines of test automation code. For quality control, these criteria must be easily accessible, reviewable and maintainable, to anyone familiar with the project.
+* We advocate complete auditability and traceability of tests cases, test results, associated specs and DUT/SUT configuration. This means a test run will record versions of every item including GitHub Repo commit SHA ids, branches, tags, SW versions, API versions, etc.
+* Clear, concise and to the point human-readable reports, plus machine-readable results allowing dashboards, rolling-up of results, etc.

documentation/dataplane/README.md

@@ -0,0 +1,11 @@
+[ [ << Back to DASH top-level Documents](../README.md#contents) ]
+
+# DASH Dataplane Documents
+This folder contains DASH dataplane design and requirements documents.
+
+# Contents
+
+| Folder                                                 | Description                                  |
+| ------------------------------------------------------ | -------------------------------------------- |
+| [design](design/README.md)                             | DASH Dataplane design & architecture documents |
+| [requirements](requirements/README.md)                 | DASH Dataplane requirements documents         |

documentation/dataplane/design/README.md

@@ -0,0 +1,12 @@
+[ [ << Back to parent directory](../README.md) ]
+
+[ [ << Back to DASH top-level Documents](../../README.md#contents) ]
+
+# DASH Dataplane Design Documents
+
+This folder contains DASH dataplane design and architecture documents.
+
+# Contents
+
+| Document                                               | Description                                |
+| ------------------------------------------------------ | ------------------------------------------ |

documentation/dataplane/requirements/README.md

@@ -0,0 +1,12 @@
+[ [ << Back to parent directory](../README.md) ]
+
+[ [ << Back to DASH top-level Documents](../../README.md#contents) ]
+
+# DASH Dataplane Requirements Documents
+
+This folder contains DASH dataplane compliance requirements documents.
+
+# Contents
+
+| Document                                               | Description                                |
+| ------------------------------------------------------ | ------------------------------------------ |

documentation/encrypt-gw-service/README.md

@@ -0,0 +1,10 @@
+[ [ << Back to DASH top-level Documents](../README.md#contents) ]
+
+This folder contains DASH Encryption Gateway Service design and requirements documents.
+
+# Contents
+
+| Folder                                                 | Description                                  |
+| ------------------------------------------------------ | -------------------------------------------- |
+| [design](design/README.md)                             | DASH Encryption Gateway Service design & architecture documents |
+| [requirements](requirements/README.md)                 | DASH Encryption Gateway Service requirements documents         |

documentation/encrypt-gw-service/design/README.md

@@ -0,0 +1,10 @@
+[ [ << Back to parent directory](../README.md) ]
+
+[ [ << Back to DASH top-level Documents](../../README.md#contents) ]
+
+This folder contains DASH Encryption Gateway Service design and architecture documents.
+
+# Contents
+
+| Document                                               | Description                                |
+| ------------------------------------------------------ | ------------------------------------------ |

documentation/encrypt-gw-service/requirements/README.md

@@ -0,0 +1,10 @@
+[ [ << Back to parent directory](../README.md) ]
+
+[ [ << Back to DASH top-level Documents](../../README.md#contents) ]
+
+This folder contains DASH Encryption Gateway Service compliance requirements documents.
+
+# Contents
+
+| Document                                               | Description                                |
+| ------------------------------------------------------ | ------------------------------------------ |

documentation/express-route-service/README.md

@@ -0,0 +1,10 @@
+[ [ << Back to DASH top-level Documents](../README.md#contents) ]
+
+This folder contains DASH Express Route Gateway Service design and requirements documents.
+
+# Contents
+
+| Folder                                                 | Description                                  |
+| ------------------------------------------------------ | -------------------------------------------- |
+| [design](design/README.md)                             | DASH Express Route Gateway Service design & architecture documents |
+| [requirements](requirements/README.md)                 | DASH Express Route Gateway Service requirements documents         |

documentation/express-route-service/design/README.md

@@ -0,0 +1,10 @@
+[ [ << Back to parent directory](../README.md) ]
+
+[ [ << Back to DASH top-level Documents](../../README.md#contents) ]
+
+This folder contains DASH Express Route Gateway Service design and architecture documents.
+
+# Contents
+
+| Document                                               | Description                                |
+| ------------------------------------------------------ | ------------------------------------------ |

documentation/express-route-service/requirements/README.md

@@ -0,0 +1,10 @@
+[ [ << Back to parent directory](../README.md) ]
+
+[ [ << Back to DASH top-level Documents](../../README.md#contents) ]
+
+This folder contains DASH Express Route Gateway Service compliance requirements documents.
+
+# Contents
+
+| Document                                               | Description                                |
+| ------------------------------------------------------ | ------------------------------------------ |

documentation/general/README.md

@@ -0,0 +1,12 @@
+[ [ << Back to DASH top-level Documents](../README.md#contents) ]
+
+# DASH General Documents
+
+This folder contains DASH "General" design and requirements documents.
+
+# Contents
+
+| Folder                                                 | Description                                  |
+| ------------------------------------------------------ | -------------------------------------------- |
+| [design](design/README.md)                             | DASH General design & architecture documents |
+| [requirements](requirements/README.md)                 | DASH General requirements documents         |

documentation/general/design/README.md

@@ -0,0 +1,14 @@
+[ [ << Back to parent directory](../README.md) ]
+
+[ [ << Back to DASH top-level Documents](../../README.md#contents) ]
+
+# DASH General Design Documents
+
+This folder contains DASH "general" design and architecture documents.
+
+# Contents
+
+| Document                                               | Description                                |
+| ------------------------------------------------------ | ------------------------------------------ |
+| [dash-high-level-design.md](dash-high-level-design.md) | DASH High-Level Architecture and Design |
+| [sdn-features-packet-transforms.md](sdn-features-packet-transforms.md) | DASH VNET-to-VNET HLD and Packet Transforms   |