[C++20][Modules] Build module static initializers per P1874R1.

Currently we only implement this for the Itanium ABI since the correct
mangling for the initializers in other ABIs is not yet known.

Intended result:

For a module interface [which includes partition interface and implementation
units] (instead of the generic CXX initializer) we emit a module init that:

 - wraps the contained initializations in a control variable to ensure that
   the inits only happen once, even if a module is imported many times by
   imports of the main unit.

 - calls module initializers for imported modules first.  Note that the
   order of module import is not significant, and therefore neither is the
   order of imported module initializers.

 - We then call initializers for the Global Module Fragment (if present)
 - We then call initializers for the current module.
 - We then call initializers for the Private Module Fragment (if present)

For a module implementation unit, or a non-module TU that imports at least one
module we emit a regular CXX init that:

 - Calls the initializers for any imported modules first.
 - Then proceeds as normal with remaining inits.

For all module unit kinds we include a global constructor entry, this allows
for the (in most cases unusual) possibility that a module object could be
included in a final binary without a specific call to its initializer.

Implementation:

 - We provide the module pointer in the AST Context so that CodeGen can act
   on it and its sub-modules.

 - We need to account for module build lines like this:
  ` clang -cc1 -std=c++20 Foo.pcm -emit-obj -o Foo.o` or
  ` clang -cc1 -std=c++20 -xc++-module Foo.cpp -emit-obj -o Foo.o`

 - in order to do this, we add to ParseAST to set the module pointer in
   the ASTContext, once we establish that this is a module build and we
   know the module pointer. To be able to do this, we make the query for
   current module public in Sema.

 - In CodeGen, we determine if the current build requires a CXX20-style module
   init and, if so, we defer any module initializers during the "Eagerly
   Emitted" phase.

 - We then walk the module initializers at the end of the TU but before
   emitting deferred inits (which adds any hidden and static ones, fixing
   llvm#51873 ).

 - We then proceed to emit the deferred inits and continue to emit the CXX
   init function.

Differential Revision: https://reviews.llvm.org/D126189

clang/include/clang/AST/ASTContext.h

@@ -472,6 +472,9 @@ class ASTContext : public RefCountedBase<ASTContext> {
   };
   llvm::DenseMap<Module*, PerModuleInitializers*> ModuleInitializers;
 
+  /// For module code-gen cases, this is the top-level module we are building.
+  Module *TopLevelModule = nullptr;
+
   static constexpr unsigned ConstantArrayTypesLog2InitSize = 8;
   static constexpr unsigned GeneralTypesLog2InitSize = 9;
   static constexpr unsigned FunctionProtoTypesLog2InitSize = 12;
@@ -1075,6 +1078,12 @@ class ASTContext : public RefCountedBase<ASTContext> {
   /// Get the initializations to perform when importing a module, if any.
   ArrayRef<Decl*> getModuleInitializers(Module *M);
 
+  /// Set the (C++20) module we are building.
+  void setModuleForCodeGen(Module *M) { TopLevelModule = M; }
+
+  /// Get module under construction, nullptr if this is not a C++20 module.
+  Module *getModuleForCodeGen() const { return TopLevelModule; }
+
   TranslationUnitDecl *getTranslationUnitDecl() const {
     return TUDecl->getMostRecentDecl();
   }

clang/include/clang/Basic/Module.h

@@ -665,6 +665,18 @@ class Module {
   Module *findSubmodule(StringRef Name) const;
   Module *findOrInferSubmodule(StringRef Name);
 
+  /// Get the Global Module Fragment (sub-module) for this module, it there is
+  /// one.
+  ///
+  /// \returns The GMF sub-module if found, or NULL otherwise.
+  Module *getGlobalModuleFragment() { return findSubmodule("<global>"); }
+
+  /// Get the Private Module Fragment (sub-module) for this module, it there is
+  /// one.
+  ///
+  /// \returns The PMF sub-module if found, or NULL otherwise.
+  Module *getPrivateModuleFragment() { return findSubmodule("<private>"); }
+
   /// Determine whether the specified module would be visible to
   /// a lookup at the end of this module.
   ///

clang/include/clang/Sema/Sema.h

@@ -2281,6 +2281,11 @@ class Sema final {
     return ModuleScopes.empty() ? nullptr : ModuleScopes.back().Module;
   }
 
+  /// Is the module scope we are an interface?
+  bool currentModuleIsInterface() const {
+    return ModuleScopes.empty() ? false : ModuleScopes.back().ModuleInterface;
+  }
+
   /// Get the module owning an entity.
   Module *getOwningModule(const Decl *Entity) {
     return Entity->getOwningModule();

clang/lib/CodeGen/CGDeclCXX.cpp

@@ -618,6 +618,127 @@ void CodeGenModule::EmitCXXThreadLocalInitFunc() {
   CXXThreadLocals.clear();
 }
 
+/* Build the initializer for a C++20 module:
+   This is arranged to be run only once regardless of how many times the module
+   might be included transitively.  This arranged by using a control variable.
+
+   First we call any initializers for imported modules.
+   We then call initializers for the Global Module Fragment (if present)
+   We then call initializers for the current module.
+   We then call initializers for the Private Module Fragment (if present)
+*/
+
+void CodeGenModule::EmitCXXModuleInitFunc(Module *Primary) {
+  while (!CXXGlobalInits.empty() && !CXXGlobalInits.back())
+    CXXGlobalInits.pop_back();
+
+  // We create the function, even if it is empty, since an importer of this
+  // module will refer to it unconditionally (for the current implementation
+  // there is no way for the importer to know that an importee does not need
+  // an initializer to be run).
+
+  // Module initializers for imported modules are emitted first.
+  // Collect the modules that we import
+  SmallVector<Module *> AllImports;
+  // Ones that we export
+  for (auto I : Primary->Exports)
+    AllImports.push_back(I.getPointer());
+  // Ones that we only import.
+  for (Module *M : Primary->Imports)
+    AllImports.push_back(M);
+
+  SmallVector<llvm::Function *, 8> ModuleInits;
+  for (Module *M : AllImports) {
+    llvm::FunctionType *FTy = llvm::FunctionType::get(VoidTy, false);
+    SmallString<256> FnName;
+    {
+      llvm::raw_svector_ostream Out(FnName);
+      cast<ItaniumMangleContext>(getCXXABI().getMangleContext())
+          .mangleModuleInitializer(M, Out);
+    }
+    assert(!GetGlobalValue(FnName.str()) &&
+           "We should only have one use of the initializer call");
+    llvm::Function *Fn = llvm::Function::Create(
+        FTy, llvm::Function::ExternalLinkage, FnName.str(), &getModule());
+    ModuleInits.push_back(Fn);
+  }
+  AllImports.clear();
+
+  // Add any initializers with specified priority; this uses the same  approach
+  // as EmitCXXGlobalInitFunc().
+  if (!PrioritizedCXXGlobalInits.empty()) {
+    SmallVector<llvm::Function *, 8> LocalCXXGlobalInits;
+    llvm::array_pod_sort(PrioritizedCXXGlobalInits.begin(),
+                         PrioritizedCXXGlobalInits.end());
+    for (SmallVectorImpl<GlobalInitData>::iterator
+             I = PrioritizedCXXGlobalInits.begin(),
+             E = PrioritizedCXXGlobalInits.end();
+         I != E;) {
+      SmallVectorImpl<GlobalInitData>::iterator PrioE =
+          std::upper_bound(I + 1, E, *I, GlobalInitPriorityCmp());
+
+      for (; I < PrioE; ++I)
+        ModuleInits.push_back(I->second);
+    }
+    PrioritizedCXXGlobalInits.clear();
+  }
+
+  // Now append the ones without specified priority.
+  for (auto F : CXXGlobalInits)
+    ModuleInits.push_back(F);
+  CXXGlobalInits.clear();
+
+  llvm::FunctionType *FTy = llvm::FunctionType::get(VoidTy, false);
+  const CGFunctionInfo &FI = getTypes().arrangeNullaryFunction();
+
+  // We now build the initializer for this module, which has a mangled name
+  // as per the Itanium ABI .  The action of the initializer is guarded so that
+  // each init is run just once (even though a module might be imported
+  // multiple times via nested use).
+  llvm::Function *Fn;
+  llvm::GlobalVariable *Guard = nullptr;
+  {
+    SmallString<256> InitFnName;
+    llvm::raw_svector_ostream Out(InitFnName);
+    cast<ItaniumMangleContext>(getCXXABI().getMangleContext())
+        .mangleModuleInitializer(Primary, Out);
+    Fn = CreateGlobalInitOrCleanUpFunction(
+        FTy, llvm::Twine(InitFnName), FI, SourceLocation(), false,
+        llvm::GlobalVariable::ExternalLinkage);
+
+    Guard = new llvm::GlobalVariable(getModule(), Int8Ty, /*isConstant=*/false,
+                                     llvm::GlobalVariable::InternalLinkage,
+                                     llvm::ConstantInt::get(Int8Ty, 0),
+                                     InitFnName.str() + "__in_chrg");
+  }
+  CharUnits GuardAlign = CharUnits::One();
+  Guard->setAlignment(GuardAlign.getAsAlign());
+
+  CodeGenFunction(*this).GenerateCXXGlobalInitFunc(
+      Fn, ModuleInits, ConstantAddress(Guard, Int8Ty, GuardAlign));
+  // We allow for the case that a module object is added to  a linked binary
+  // without a specific call to the the initializer.  This also ensure that
+  // implementation partition initializers are called when the partition
+  // is not imported as an interface.
+  AddGlobalCtor(Fn);
+
+  // See the comment in EmitCXXGlobalInitFunc about OpenCL global init
+  // functions.
+  if (getLangOpts().OpenCL) {
+    GenKernelArgMetadata(Fn);
+    Fn->setCallingConv(llvm::CallingConv::SPIR_KERNEL);
+  }
+
+  assert(!getLangOpts().CUDA || !getLangOpts().CUDAIsDevice ||
+         getLangOpts().GPUAllowDeviceInit);
+  if (getLangOpts().HIP && getLangOpts().CUDAIsDevice) {
+    Fn->setCallingConv(llvm::CallingConv::AMDGPU_KERNEL);
+    Fn->addFnAttr("device-init");
+  }
+
+  ModuleInits.clear();
+}
+
 static SmallString<128> getTransformedFileName(llvm::Module &M) {
   SmallString<128> FileName = llvm::sys::path::filename(M.getName());
 
@@ -650,7 +771,26 @@ CodeGenModule::EmitCXXGlobalInitFunc() {
   while (!CXXGlobalInits.empty() && !CXXGlobalInits.back())
     CXXGlobalInits.pop_back();
 
-  if (CXXGlobalInits.empty() && PrioritizedCXXGlobalInits.empty())
+  // When we import C++20 modules, we must run their initializers first.
+  SmallVector<llvm::Function *, 8> ModuleInits;
+  if (CXX20ModuleInits)
+    for (Module *M : ImportedModules) {
+      llvm::FunctionType *FTy = llvm::FunctionType::get(VoidTy, false);
+      SmallString<256> FnName;
+      {
+        llvm::raw_svector_ostream Out(FnName);
+        cast<ItaniumMangleContext>(getCXXABI().getMangleContext())
+            .mangleModuleInitializer(M, Out);
+      }
+      assert(!GetGlobalValue(FnName.str()) &&
+             "We should only have one use of the initializer call");
+      llvm::Function *Fn = llvm::Function::Create(
+          FTy, llvm::Function::ExternalLinkage, FnName.str(), &getModule());
+      ModuleInits.push_back(Fn);
+    }
+
+  if (ModuleInits.empty() && CXXGlobalInits.empty() &&
+      PrioritizedCXXGlobalInits.empty())
     return;
 
   llvm::FunctionType *FTy = llvm::FunctionType::get(VoidTy, false);
@@ -676,6 +816,13 @@ CodeGenModule::EmitCXXGlobalInitFunc() {
       llvm::Function *Fn = CreateGlobalInitOrCleanUpFunction(
           FTy, "_GLOBAL__I_" + getPrioritySuffix(Priority), FI);
 
+      // Prepend the module inits to the highest priority set.
+      if (!ModuleInits.empty()) {
+        for (auto F : ModuleInits)
+          LocalCXXGlobalInits.push_back(F);
+        ModuleInits.clear();
+      }
+
       for (; I < PrioE; ++I)
         LocalCXXGlobalInits.push_back(I->second);
 
@@ -685,17 +832,33 @@ CodeGenModule::EmitCXXGlobalInitFunc() {
     PrioritizedCXXGlobalInits.clear();
   }
 
-  if (getCXXABI().useSinitAndSterm() && CXXGlobalInits.empty())
+  if (getCXXABI().useSinitAndSterm() && ModuleInits.empty() &&
+      CXXGlobalInits.empty())
     return;
 
+  for (auto F : CXXGlobalInits)
+    ModuleInits.push_back(F);
+  CXXGlobalInits.clear();
+
   // Include the filename in the symbol name. Including "sub_" matches gcc
   // and makes sure these symbols appear lexicographically behind the symbols
   // with priority emitted above.
-  llvm::Function *Fn = CreateGlobalInitOrCleanUpFunction(
-      FTy, llvm::Twine("_GLOBAL__sub_I_", getTransformedFileName(getModule())),
-      FI);
-
-  CodeGenFunction(*this).GenerateCXXGlobalInitFunc(Fn, CXXGlobalInits);
+  llvm::Function *Fn;
+  if (CXX20ModuleInits && getContext().getModuleForCodeGen()) {
+    SmallString<256> InitFnName;
+    llvm::raw_svector_ostream Out(InitFnName);
+    cast<ItaniumMangleContext>(getCXXABI().getMangleContext())
+        .mangleModuleInitializer(getContext().getModuleForCodeGen(), Out);
+    Fn = CreateGlobalInitOrCleanUpFunction(
+        FTy, llvm::Twine(InitFnName), FI, SourceLocation(), false,
+        llvm::GlobalVariable::ExternalLinkage);
+  } else
+    Fn = CreateGlobalInitOrCleanUpFunction(
+        FTy,
+        llvm::Twine("_GLOBAL__sub_I_", getTransformedFileName(getModule())),
+        FI);
+
+  CodeGenFunction(*this).GenerateCXXGlobalInitFunc(Fn, ModuleInits);
   AddGlobalCtor(Fn);
 
   // In OpenCL global init functions must be converted to kernels in order to
@@ -718,7 +881,7 @@ CodeGenModule::EmitCXXGlobalInitFunc() {
     Fn->addFnAttr("device-init");
   }
 
-  CXXGlobalInits.clear();
+  ModuleInits.clear();
 }
 
 void CodeGenModule::EmitCXXGlobalCleanUpFunc() {

clang/lib/CodeGen/CodeGenModule.cpp

@@ -136,6 +136,13 @@ CodeGenModule::CodeGenModule(ASTContext &C, const HeaderSearchOptions &HSO,
   GlobalsInt8PtrTy = Int8Ty->getPointerTo(DL.getDefaultGlobalsAddressSpace());
   ASTAllocaAddressSpace = getTargetCodeGenInfo().getASTAllocaAddressSpace();
 
+  // Build C++20 Module initializers.
+  // TODO: Add Microsoft here once we know the mangling required for the
+  // initializers.
+  CXX20ModuleInits =
+      LangOpts.CPlusPlusModules && getCXXABI().getMangleContext().getKind() ==
+                                       ItaniumMangleContext::MK_Itanium;
+
   RuntimeCC = getTargetCodeGenInfo().getABIInfo().getRuntimeCC();
 
   if (LangOpts.ObjC)
@@ -509,12 +516,18 @@ static void setVisibilityFromDLLStorageClass(const clang::LangOptions &LO,
 }
 
 void CodeGenModule::Release() {
+  Module *Primary = getContext().getModuleForCodeGen();
+  if (CXX20ModuleInits && Primary)
+    EmitModuleInitializers(Primary);
   EmitDeferred();
   EmitVTablesOpportunistically();
   applyGlobalValReplacements();
   applyReplacements();
   emitMultiVersionFunctions();
-  EmitCXXGlobalInitFunc();
+  if (CXX20ModuleInits && Primary && Primary->isInterfaceOrPartition())
+    EmitCXXModuleInitFunc(Primary);
+  else
+    EmitCXXGlobalInitFunc();
   EmitCXXGlobalCleanUpFunc();
   registerGlobalDtorsWithAtExit();
   EmitCXXThreadLocalInitFunc();
@@ -2491,6 +2504,31 @@ static void addLinkOptionsPostorder(CodeGenModule &CGM, Module *Mod,
   }
 }
 
+void CodeGenModule::EmitModuleInitializers(clang::Module *Primary) {
+  // Emit the initializers in the order that sub-modules appear in the
+  // source, first Global Module Fragments, if present.
+  if (auto GMF = Primary->getGlobalModuleFragment()) {
+    for (Decl *D : getContext().getModuleInitializers(GMF)) {
+      assert(D->getKind() == Decl::Var && "GMF initializer decl is not a var?");
+      EmitTopLevelDecl(D);
+    }
+  }
+  // Second any associated with the module, itself.
+  for (Decl *D : getContext().getModuleInitializers(Primary)) {
+    // Skip import decls, the inits for those are called explicitly.
+    if (D->getKind() == Decl::Import)
+      continue;
+    EmitTopLevelDecl(D);
+  }
+  // Third any associated with the Privat eMOdule Fragment, if present.
+  if (auto PMF = Primary->getPrivateModuleFragment()) {
+    for (Decl *D : getContext().getModuleInitializers(PMF)) {
+      assert(D->getKind() == Decl::Var && "PMF initializer decl is not a var?");
+      EmitTopLevelDecl(D);
+    }
+  }
+}
+
 void CodeGenModule::EmitModuleLinkOptions() {
   // Collect the set of all of the modules we want to visit to emit link
   // options, which is essentially the imported modules and all of their
@@ -2896,12 +2934,19 @@ bool CodeGenModule::MayBeEmittedEagerly(const ValueDecl *Global) {
       // explicitly instantiated, so they should not be emitted eagerly.
       return false;
   }
-  if (const auto *VD = dyn_cast<VarDecl>(Global))
+  if (const auto *VD = dyn_cast<VarDecl>(Global)) {
     if (Context.getInlineVariableDefinitionKind(VD) ==
         ASTContext::InlineVariableDefinitionKind::WeakUnknown)
       // A definition of an inline constexpr static data member may change
       // linkage later if it's redeclared outside the class.
       return false;
+    if (CXX20ModuleInits && VD->getOwningModule()) {
+      // For CXX20, module-owned initializers need to be deferred, since it is
+      // not known at this point if they will be run for the current module or
+      // as part of the initializer for an imported one.
+      return false;
+    }
+  }
   // If OpenMP is enabled and threadprivates must be generated like TLS, delay
   // codegen for global variables, because they may be marked as threadprivate.
   if (LangOpts.OpenMP && LangOpts.OpenMPUseTLS &&
@@ -6198,6 +6243,12 @@ void CodeGenModule::EmitTopLevelDecl(Decl *D) {
         DI->EmitImportDecl(*Import);
     }
 
+    // For CXX20 we are done - we will call the module initializer for the
+    // imported module, and that will likewise call those for any imports it
+    // has.
+    if (CXX20ModuleInits)
+      break;
+
     // Find all of the submodules and emit the module initializers.
     llvm::SmallPtrSet<clang::Module *, 16> Visited;
     SmallVector<clang::Module *, 16> Stack;