Skip to content

Commit

Permalink
refactor: Only publish single aws-pdk package (aws#536)
Browse files Browse the repository at this point in the history
This commit restructures the PDK as follows:

- Individual packages are now namespaced as @aws-pdk and are NOT published to npm/pypi/maven.
- A new aws-pdk package is created which creates a single distributable which IS published to npm/pypi/maven.
- Dependencies between packages is supported via @aws-pdk/XXX. When packaging into the aws-pdk, these imports are rewritten using a combination of tsconfig paths and tsc-alias.
- PDKProject has been refactored to pull out PDKRelease as this should now only happen in AwsPdkProject.
- Package tasks have been removed from individual packages as this now should only occur in aws-pdk.
- aws_arch and cdk_graph_plugin_diagram have been refactored slightly to ensure exported types make JSII happy.
- Overall DAG has been simplified and improvements to build times have been made ~8 mins
- docs has been pulled up to the root level
- private has been replaces by projenrc
- PDKPipelineXXProject has been removed from the pipeline package.
- Updated Monorepo to ensure NodeResolution is nodenext for TS projects
- Changed transient deps script to use shallowest transient dep version instead of greatest given we are now bundling everything together in one.
- Interfaces no longer expose required props when they aren't actually required.
- All TSProjects under a monorepo use tsconfig module reoslution settings of nodenext and target es2021
- Added a global pdk command which aliases underlying package manager command 🕺
- Added an API Explorer powered by SwaggerUI to the CloudscapeReactTsWebsite which allows you to make authenticated API requests
- Added an Infra package which allows you to create boilerplate CDK code for an API + Website. This means you can get a fully functioning website + API, with all Infra in < 70 LoC 👏

BREAKING CHANGES
- All imports will need to be changed to aws-pdk/<pkg>
- All dependencies on @aws-prototyping-sdk/* will need to be changed to a single dependency on aws-pdk
- Refactored CDKGraph & CdkGraphDiagramPlugin to poly friendly definitions which will break existing consumers in TS (Py and Java never worked).
- Removed PDKPipeline projen constructs.
- Renamed nx-monorepo to monorepo
- TypeSafeApi Python runtime moved to new python-nextgen OpenAPI generator, which means:
  - Properties of models can be referenced as attributes rather than dictionary syntax (eg .my_property rather than ["my_property"]).
  - Models are now serialised and deserialised using the .from_json and .to_json methods.
  - request_parameters and request_array_parameters are also models.
- TypeSafeApi Python handlers import location changed from <package>.apis.tags.default_api_operation_config to <package>.api.operation_config.
- TypeSafeApi Java classes for handlers/interceptors in runtime are no longer static subclasses of the Handlers class. The import location has changed from eg <package>.api.Handlers.SayHello to <package>.api.handlers.say_hello.SayHello.
  • Loading branch information
agdimech authored Aug 27, 2023
1 parent 1fcf948 commit c43f3b6
Show file tree
Hide file tree
Showing 607 changed files with 34,238 additions and 573,434 deletions.
26 changes: 13 additions & 13 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ jobs:
- name: Derive appropriate SHAs for base and head for `nx affected` commands
uses: nrwl/nx-set-shas@v2
with:
main-branch-name: 'mainline'
main-branch-name: "mainline"
- name: Set git identity
run: |-
git config user.name "github-actions"
Expand All @@ -42,7 +42,7 @@ jobs:
id: git_remote
run: echo ::set-output name=latest_commit::"$(git ls-remote origin -h ${{ github.ref }} | cut -f1)"
- name: Extract Dists
run: rsync -a . ./dist --include="*/" --include="/public/docs/dist/**" --include="/packages/*/dist/**" --exclude="*" --prune-empty-dirs
run: rsync -a . ./dist --include="*/" --include="/docs/dist/**" --include="/packages/aws-pdk/dist/**" --exclude="*" --prune-empty-dirs
- name: Upload artifact
if: ${{ steps.git_remote.outputs.latest_commit == github.sha }}
uses: actions/[email protected]
Expand All @@ -67,7 +67,7 @@ jobs:
path: dist
- name: Release
run: errout=$(mktemp); gh release create $(cat dist/releasetag.txt) -R $GITHUB_REPOSITORY -F dist/changelog.md -t $(cat dist/releasetag.txt) --target $GITHUB_REF 2> $errout && true; exitcode=$?; if [ $exitcode -ne 0 ] && ! grep -q "Release.tag_name already exists" $errout; then cat $errout; exit $exitcode; fi
working-directory: dist/packages/nx-monorepo
working-directory: dist/packages/aws-pdk
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_REPOSITORY: ${{ github.repository }}
Expand All @@ -92,8 +92,8 @@ jobs:
name: build-artifact
path: dist
- name: Release
working-directory: dist/packages
run: for d in *; do cd $d && ( [ -d "dist/js" ] && pnpm --package publib@latest dlx publib-npm || echo "Ignore `basename $PWD` - no dist/js" ) && cd ..; done;
working-directory: dist/packages/aws-pdk
run: pnpm --package publib@latest dlx publib-npm || echo "Ignore `basename $PWD` - no dist/js"
env:
NPM_DIST_TAG: latest
NPM_REGISTRY: registry.npmjs.org
Expand Down Expand Up @@ -122,8 +122,8 @@ jobs:
name: build-artifact
path: dist
- name: Release
working-directory: dist/packages
run: for d in *; do cd $d && ( [ -d "dist/java" ] && pnpm --package publib@latest dlx publib-maven || echo "Ignore `basename $PWD` - no dist/java" ) && cd ..; done;
working-directory: dist/packages/aws-pdk
run: pnpm --package publib@latest dlx publib-maven || echo "Ignore `basename $PWD` - no dist/java"
env:
MAVEN_ENDPOINT: https://aws.oss.sonatype.org
MAVEN_GPG_PRIVATE_KEY: ${{ secrets.MAVEN_GPG_PRIVATE_KEY }}
Expand Down Expand Up @@ -154,8 +154,8 @@ jobs:
name: build-artifact
path: dist
- name: Release
working-directory: dist/packages
run: for d in *; do cd $d && ( [ -d "dist/python" ] && pnpm --package publib@latest dlx publib-pypi || echo "Ignore `basename $PWD` - no dist/python" ) && cd ..; done;
working-directory: dist/packages/aws-pdk
run: pnpm --package publib@latest dlx publib-pypi || echo "Ignore `basename $PWD` - no dist/python"
env:
TWINE_USERNAME: ${{ secrets.TWINE_USERNAME }}
TWINE_PASSWORD: ${{ secrets.TWINE_PASSWORD }}
Expand All @@ -181,16 +181,16 @@ jobs:
git config user.name "AWS PDK Automation"
git config user.email "[email protected]"
- name: Upload docs to Github
run: zip -r docs.zip dist/public/docs/dist/docs/* && gh release upload $(cat dist/packages/nx-monorepo/dist/releasetag.txt) -R $GITHUB_REPOSITORY docs.zip && rm docs.zip
run: zip -r docs.zip dist/docs/dist/docs/* && gh release upload $(cat dist/packages/aws-pdk/dist/releasetag.txt) -R $GITHUB_REPOSITORY docs.zip && rm docs.zip
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_REPOSITORY: ${{ github.repository }}
- name: Prepare Commit
run: |-
mv dist/public/docs/dist ${{ runner.temp }}/dist
mv dist/docs/dist ${{ runner.temp }}/dist
rsync --delete --exclude=.git --recursive ${{ runner.temp }}/dist/docs/ .
touch .nojekyll
git add .
git diff --cached --exit-code >/dev/null || (git commit -am 'docs: publish from ${{ github.sha }}')
- name: Push
run: git push origin gh-pages:gh-pages
# - name: Push
# run: git push origin gh-pages:gh-pages
17 changes: 15 additions & 2 deletions .projen/deps.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

40 changes: 8 additions & 32 deletions .projen/tasks.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

44 changes: 20 additions & 24 deletions .projenrc.ts
Original file line number Diff line number Diff line change
@@ -1,45 +1,41 @@
/*! Copyright [Amazon.com](http://amazon.com/), Inc. or its affiliates. All Rights Reserved.
SPDX-License-Identifier: Apache-2.0 */
import { NxProject } from "./packages/nx-monorepo/src/components/nx-project";
import { PDKProject } from "./private/pdk-project";
import { AwsArchProject } from "./private/projects/aws-arch";
import { CdkGraphProject } from "./private/projects/cdk-graph";
import { CdkGraphPluginDiagramProject } from "./private/projects/cdk-graph-plugin-diagram";
import { CloudscapeReactTsWebsiteProject } from "./private/projects/cloudscape-react-ts-website";
import { DocsProject } from "./private/projects/docs-project";
import { IdentityProject } from "./private/projects/identity-project";
import { NXMonorepoProject } from "./private/projects/nx-monorepo-project";
import { OpenApiGatewayProject } from "./private/projects/open-api-gateway-project";
import { PDKMonorepoProject } from "./private/projects/pdk-monorepo-project";
import { PDKNagProject } from "./private/projects/pdk-nag-project";
import { PipelineProject } from "./private/projects/pipeline-project";
import { StaticWebsiteProject } from "./private/projects/static-website-project";
import { TypeSafeApiProject } from "./private/projects/type-safe-api-project";
import { AwsArchProject } from "./projenrc/projects/aws-arch-project";
import { AwsPdkProject } from "./projenrc/projects/aws-pdk-project";
import { CdkGraphPluginDiagramProject } from "./projenrc/projects/cdk-graph-plugin-diagram-project";
import { CdkGraphProject } from "./projenrc/projects/cdk-graph-project";
import { CloudscapeReactTsWebsiteProject } from "./projenrc/projects/cloudscape-react-ts-website-project";
import { DocsProject } from "./projenrc/projects/docs-project";
import { IdentityProject } from "./projenrc/projects/identity-project";
import { InfrastructureProject } from "./projenrc/projects/infrastructure-project";
import { MonorepoProject } from "./projenrc/projects/monorepo-project";
import { PDKMonorepoProject } from "./projenrc/projects/pdk-monorepo-project";
import { PDKNagProject } from "./projenrc/projects/pdk-nag-project";
import { PipelineProject } from "./projenrc/projects/pipeline-project";
import { StaticWebsiteProject } from "./projenrc/projects/static-website-project";
import { TypeSafeApiProject } from "./projenrc/projects/type-safe-api-project";

// root/parent project
const monorepoProject = new PDKMonorepoProject();

new PDKNagProject(monorepoProject);

// public packages
new NXMonorepoProject(monorepoProject);
new MonorepoProject(monorepoProject);
new StaticWebsiteProject(monorepoProject);
new IdentityProject(monorepoProject);
new OpenApiGatewayProject(monorepoProject);
new TypeSafeApiProject(monorepoProject);
new CloudscapeReactTsWebsiteProject(monorepoProject);
new AwsArchProject(monorepoProject);
new CdkGraphProject(monorepoProject);
new CdkGraphPluginDiagramProject(monorepoProject);
new PipelineProject(monorepoProject);
new InfrastructureProject(monorepoProject);

// This must always appear after all other packages!
new AwsPdkProject(monorepoProject);

// docs
const docsProject = new DocsProject(monorepoProject);
// Docs should have a dependency on all publishable packages
NxProject.ensure(docsProject).addImplicitDependency(
...monorepoProject.sortedSubProjects.filter(
(s: any) => s instanceof PDKProject && s.pdkRelease
)
);
new DocsProject(monorepoProject);

monorepoProject.synth();
20 changes: 11 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,13 +4,13 @@

The AWS Prototyping SDK (PDK) aims to accelerate the development of prototypes on AWS. To achieve this, it provides building blocks for common patterns together with development tools to manage and build your projects. The constructs are based on [AWS CDK](https://github.com/aws/aws-cdk) and implement an expanding number of common application/infrastructure patterns, such as the ability to generate complete client and infrastructure code for a REST API from an OpenAPI specification. The project and build tools the PDK provide allows you manage the configuration of multiple related projects in a language-agnostic way, and to efficiently execute parallel incremental builds.

This combination of project management, build execution, and CDK constructs reduce the effort involved in prototyping new ideas on AWS, and allow developers to seamlessly grow these ideas beyond the prototype phase without requiring a complete re-implementation of the original solution. In addition to the foundation of project configuration and build dependency management provided by the nx-monorepo project, PDK also provides a number of other projen projects (some of which implement common infrastructure patterns using CDK) and stand-alone CDK constructs, which are described below. Note: PDK is currently pre-release, targeting version 1.0 for mid-2023.
This combination of project management, build execution, and CDK constructs reduce the effort involved in prototyping new ideas on AWS, and allow developers to seamlessly grow these ideas beyond the prototype phase without requiring a complete re-implementation of the original solution. In addition to the foundation of project configuration and build dependency management provided by the monorepo project, PDK also provides a number of other projen projects (some of which implement common infrastructure patterns using CDK) and stand-alone CDK constructs, which are described below. Note: PDK is currently pre-release, targeting version 1.0 for mid-2023.

## Core Modules

### [nx-monorepo](./packages/nx-monorepo/README.md)
### [monorepo](./packages/monorepo/README.md)

Modern applications are often implemented across multiple packages or libraries, each potentially implemented in a different language. To help manage the complexity that arises from this, PDK provides a [Projen](https://github.com/projen/projen) project called 'nx-monorepo' to support dependency management and builds across packages. Together with the project-as-code provided by Projen, this project utilises [Nx Build](https://nx.dev/) to give developers the ability to manage a collection of different projects within a single repository. Nx Build allows for explicit and implicit dependency management between polyglot projects, and provides shared caching and parallel task execution for super fast builds across multiple languages with ease. By combining Projen and NX Build, 'nx-monorepo' reduces the effort involved in maintaining and building related projects implemented in a mix of languages.
Modern applications are often implemented across multiple packages or libraries, each potentially implemented in a different language. To help manage the complexity that arises from this, PDK provides a [Projen](https://github.com/projen/projen) project called 'monorepo' to support dependency management and builds across packages. Together with the project-as-code provided by Projen, this project utilises [Nx Build](https://nx.dev/) to give developers the ability to manage a collection of different projects within a single repository. Nx Build allows for explicit and implicit dependency management between polyglot projects, and provides shared caching and parallel task execution for super fast builds across multiple languages with ease. By combining Projen and NX Build, 'monorepo' reduces the effort involved in maintaining and building related projects implemented in a mix of languages.

### [static-website](./packages/static-website/README.md)

Expand All @@ -26,7 +26,7 @@ This module provides a projen project that allows you to define an API using eit

### [pipeline](./packages/pipeline/README.md)

This module provides a projen project that uses a construct based on CDK's CodePipeline construct, named PDKPipeline, to deploy a CI/CD pipeline. It additionally creates a CodeCommit repository and by default is configured to build the project assuming nx-monorepo is being used (although this can be changed). A Sonarqube Scanner can also be configured to trigger a scan whenever the synth build job completes successfully. This Scanner is non-blocking and as such is not instrumented as part of the pipeline.
This module provides a projen project that uses a construct based on CDK's CodePipeline construct, named PDKPipeline, to deploy a CI/CD pipeline. It additionally creates a CodeCommit repository and by default is configured to build the project assuming monorepo is being used (although this can be changed). A Sonarqube Scanner can also be configured to trigger a scan whenever the synth build job completes successfully. This Scanner is non-blocking and as such is not instrumented as part of the pipeline.

### [pdk-nag](./packages/pdk-nag/README.md)

Expand All @@ -48,10 +48,10 @@ This module provides a CDK derived construct that can be added to existing CDK p

Ensure you have the following packages installed globally:

* [pnpm](https://pnpm.io/installation)
* [node > 14](https://nodejs.org/en/download/package-manager/) (or use [nvm](https://github.com/nvm-sh/nvm#installing-and-updating) to install)
* [Python >= 3.7](https://www.python.org/downloads/)
* [Java >= 8](https://aws.amazon.com/fr/corretto/) and [Maven >= 3.6](https://maven.apache.org/download.cgi)
- [pnpm](https://pnpm.io/installation)
- [node > 14](https://nodejs.org/en/download/package-manager/) (or use [nvm](https://github.com/nvm-sh/nvm#installing-and-updating) to install)
- [Python >= 3.7](https://www.python.org/downloads/)
- [Java >= 8](https://aws.amazon.com/fr/corretto/) and [Maven >= 3.6](https://maven.apache.org/download.cgi)

```bash
# from root directory of this package
Expand All @@ -60,7 +60,7 @@ pnpm i

## Quickstart

The [README for the 'nx-monorepo'](./packages/nx-monorepo/README.md) projen project provides an example of how to get started with PDK.
The [README for the 'monorepo'](./packages/monorepo/README.md) projen project provides an example of how to get started with PDK.

## Usage of projen/nx

Expand All @@ -75,11 +75,13 @@ To build the full project, run `pnpm nx run-many --target=build --all`
Any change to `projects/*` or `.projenrc.ts` requires a synth to be executed. To do this, run: `pnpm projen` from the root directory.

## Nx workspace script alias

In addition to the above `pnpm nx <command>` format to execute commands, the workspace package contains useful alias for common tasks.

Executing `pnpm <task>` for common tasks will execute `pnpm nx run-many --target=<task> --output-style=stream --nx-bail`, such as `pnpm build` will execute `pnpx nx run-many --target=build --output-style=stream --nx-bail` across all packages.

All nx run-many alias scripts access additional arguments, such as to only run on specific projects you can use `pnpm build --projects=proj1,proj2`.

> See [Nx Run-Many options](https://nx.dev/packages/nx/documents/run-many#options) for details.
## Documentation
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
5 changes: 5 additions & 0 deletions public/docs/.projen/deps.json → docs/.projen/deps.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes
File renamed without changes
File renamed without changes.
File renamed without changes.
Loading

0 comments on commit c43f3b6

Please sign in to comment.