Skip to content

Commit

Permalink
Add -p autogen-subclasses to print all descendants of specified sup…
Browse files Browse the repository at this point in the history
…erclasses (sorbet#1040)

* Add `-p autogen-subclasses` to print all superclasses and their subclasses

Fix CLI help test

Minor refactors

Allow passes that are run after the one you need

Enable printing subclasses to output file

Add path-exclude filters

Minor refactors, remove unnecessary membership checks

Comment out filepath-excludes to see if affects perf

Ignore definitions that don't define classes

Include Definition::type in `-p subclasses` output

`InheritedClassesStep` traverses inheritance through modules (to find "class Foo which includes module Bar which includes module Baz" as a descendant of Baz) but does not include modules in SUBCLASSES (i.e., does not list "Bar" as a descendant of Baz).

Print Definition type as human-readable string

Ignore pay-server files we don't care about

...but don't ignore the Sorbet CLI test files

Move path exclusion to the right spot

* Add options for ignoring files and selecting parent classes

* Generate the equivalent of DescendantsMap

* Minor refactors

* Add and test manual overrides

* Test file ignoring

* Remove extraneous `typeAsStringView`

* Simplify options handling

autogen-subclasses doesn't actually need resolver, just namer.

* Move autogen-subclasses logic out of realmain.cc

* Don't store capturing lambdas

* Fix test

* Return stuff from ParsedFile methods instead of using &out refs

* Use UnorderedMap and UnorderedSet

(i.e., abseil flat hashes)

* Prefer emplace_back over push_back

* More `const`

* Change method names to actions instead of nouns

* Even more `const`

* Fix CLI test for `--help`

* Re-run clang format

* Remove unnecessary `const core::Context`

* Use `shortName` instead of `show`

* Pull AutogenSubclasses stuff into a new autogen::Subclasses class

* Move subclasses-related stuff to new subclasses.{h, cc}

* Destructure `Entry`s for better readability

* Cargo-cult a modded `isFileIgnored` to avoid abusing the original

* Check UnorderedMap membership more idiomatically

* Fix --autogen-subclasses-ignore test

* Test some edge cases

1) User asking for class that doesn't exist
2) User asking for class that is never subclassed

* Refactor descendantsOf to return a value

* Kill `maybeInsertChild`, which didn't need to be it's own method

* Make `serializeSubclassMap` private

* Add some comments

* Remove some TODOs

* Add trailing comma to BUILD file

* Split up `matchIsFolderOrFile` into multiple methods
  • Loading branch information
gw authored Jul 10, 2019
1 parent e2ca390 commit bbbe525
Show file tree
Hide file tree
Showing 18 changed files with 404 additions and 21 deletions.
2 changes: 2 additions & 0 deletions common/FileOps.h
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ namespace sorbet {
class FileOps final {
public:
static bool exists(std::string_view filename);
static bool isFile(std::string_view path, std::string_view ignorePattern, const int pos);
static bool isFolder(std::string_view path, std::string_view ignorePattern, const int pos);
static std::string read(std::string_view filename);
static void write(std::string_view filename, const std::vector<sorbet::u1> &data);
static void append(std::string_view filename, std::string_view text);
Expand Down
18 changes: 12 additions & 6 deletions common/common.cc
Original file line number Diff line number Diff line change
Expand Up @@ -159,11 +159,16 @@ optional<string> sorbet::FileOps::readLineFromFd(int fd, string &buffer, int tim
}
}

// Verifies that next character after the match is '/' (indicating a folder match) or end of string (indicating a file
// match).
bool matchIsFolderOrFile(string_view path, string_view ignorePattern, const int pos) {
// Verifies that a matching pattern occurs at the end of the matched path
bool sorbet::FileOps::isFile(string_view path, string_view ignorePattern, const int pos) {
const int endPos = pos + ignorePattern.length();
return endPos == path.length() || path.at(endPos) == '/';
return endPos == path.length();
}

// Verifies that a matching pattern is followed by a "/" in the matched path
bool sorbet::FileOps::isFolder(string_view path, string_view ignorePattern, const int pos) {
const int endPos = pos + ignorePattern.length();
return path.at(endPos) == '/';
}

// Simple, naive implementation of regexp-free ignore rules.
Expand All @@ -174,7 +179,8 @@ bool sorbet::FileOps::isFileIgnored(string_view basePath, string_view filePath,
// Note: relative_path always includes a leading /
string_view relative_path = filePath.substr(basePath.length());
for (auto &p : absoluteIgnorePatterns) {
if (relative_path.substr(0, p.length()) == p && matchIsFolderOrFile(relative_path, p, 0)) {
if (relative_path.substr(0, p.length()) == p &&
(isFile(relative_path, p, 0) || isFolder(relative_path, p, 0))) {
return true;
}
}
Expand All @@ -185,7 +191,7 @@ bool sorbet::FileOps::isFileIgnored(string_view basePath, string_view filePath,
pos = relative_path.find(p, pos);
if (pos == string_view::npos) {
break;
} else if (matchIsFolderOrFile(relative_path, p, pos)) {
} else if (isFile(relative_path, p, pos) || isFolder(relative_path, p, pos)) {
return true;
}
pos += p.length();
Expand Down
2 changes: 2 additions & 0 deletions main/autogen/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,11 @@ cc_library(
name = "autogen",
srcs = [
"autogen.cc",
"subclasses.cc",
],
hdrs = [
"autogen.h",
"subclasses.h",
],
linkstatic = select({
"//tools/config:linkshared": 0,
Expand Down
13 changes: 9 additions & 4 deletions main/autogen/autogen.cc
Original file line number Diff line number Diff line change
Expand Up @@ -663,15 +663,20 @@ string ParsedFile::toMsgpack(core::Context ctx, int version) {
return write.pack(ctx, *this);
}

void ParsedFile::classlist(core::Context ctx, vector<string> &out) {
auto nameToString = [&](const auto &nm) -> string { return nm.data(ctx)->show(ctx); };
vector<string> ParsedFile::listAllClasses(core::Context ctx) {
vector<string> out;

for (auto &def : defs) {
if (def.type != Definition::Class) {
continue;
}
auto names = showFullName(ctx, def.id);
out.emplace_back(fmt::format("{}", fmt::map_join(names, "::", nameToString)));
vector<core::NameRef> names = showFullName(ctx, def.id);
out.emplace_back(fmt::format("{}", fmt::map_join(names, "::", [&ctx](const core::NameRef &nm) -> string_view {
return nm.data(ctx)->shortName(ctx);
})));
}

return out;
}

} // namespace sorbet::autogen
10 changes: 6 additions & 4 deletions main/autogen/autogen.h
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
#ifndef AUTOGEN_H
#define AUTOGEN_H
#include "ast/ast.h"

namespace sorbet::autogen {
Expand Down Expand Up @@ -72,6 +74,8 @@ struct Reference {
};

struct ParsedFile {
friend class MsgpackWriter;

ast::ParsedFile tree;
u4 cksum;
std::string path;
Expand All @@ -81,11 +85,8 @@ struct ParsedFile {

std::string toString(core::Context ctx);
std::string toMsgpack(core::Context ctx, int version);
void classlist(core::Context ctx, std::vector<std::string> &out);

private:
std::vector<core::NameRef> showFullName(core::Context ctx, DefinitionRef id);
friend class MsgpackWriter;
std::vector<std::string> listAllClasses(core::Context ctx);
};

class Autogen final {
Expand All @@ -94,3 +95,4 @@ class Autogen final {
Autogen() = delete;
};
} // namespace sorbet::autogen
#endif // AUTOGEN_H
167 changes: 167 additions & 0 deletions main/autogen/subclasses.cc
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
#include "main/autogen/subclasses.h"
#include "common/FileOps.h"

using namespace std;
namespace sorbet::autogen {

// Analogue of sorbet::FileOps::isFileIgnored that doesn't take a basePath, since
// we don't need one here, and using FileOps' version meant passing some weird,
// hard-to-understand arguments to mimic how other callers use it.
bool Subclasses::isFileIgnored(const std::string &path, const std::vector<std::string> &absoluteIgnorePatterns,
const std::vector<std::string> &relativeIgnorePatterns) {
for (auto &p : absoluteIgnorePatterns) {
if (path.substr(0, p.length()) == p &&
(sorbet::FileOps::isFile(path, p, 0) || sorbet::FileOps::isFolder(path, p, 0))) {
return true;
}
}
for (auto &p : relativeIgnorePatterns) {
// See if /pattern is in string, and that it matches a whole folder or file name.
int pos = 0;
while (true) {
pos = path.find(p, pos);
if (pos == string_view::npos) {
break;
} else if (sorbet::FileOps::isFile(path, p, pos) || sorbet::FileOps::isFolder(path, p, pos)) {
return true;
}
pos += p.length();
}
}
return false;
};

// Get all subclasses defined in a particular ParsedFile
optional<Subclasses::Map> Subclasses::listAllSubclasses(core::Context ctx, ParsedFile &pf,
const vector<string> &absoluteIgnorePatterns,
const vector<string> &relativeIgnorePatterns) {
// Prepend "/" because absoluteIgnorePatterns and relativeIgnorePatterns are always "/"-prefixed
if (isFileIgnored(fmt::format("/{}", pf.path), absoluteIgnorePatterns, relativeIgnorePatterns)) {
return nullopt;
}

Subclasses::Map out;
for (const Reference &ref : pf.refs) {
DefinitionRef defn = ref.parent_of;
if (!defn.exists()) {
// This is just a random constant reference and doesn't
// define a Child < Parent relationship.
continue;
}

// Get fully-qualified parent name as string
string parentName =
fmt::format("{}", fmt::map_join(ref.resolved, "::", [&ctx](const core::NameRef &nm) -> string {
return nm.data(ctx)->show(ctx);
}));

// Add child class to the set identified by its parent
string childName = fmt::format(
"{}", fmt::map_join(pf.showFullName(ctx, defn),
"::", [&ctx](const core::NameRef &nm) -> string { return nm.data(ctx)->show(ctx); }));

out[parentName].insert(make_pair(childName, defn.data(pf).type));
}

return out;
}

// Generate all descendants of a parent class
// Recursively walks `childMap`, which stores the IMMEDIATE children of subclassed class.
optional<Subclasses::Entries> Subclasses::descendantsOf(const Subclasses::Map &childMap, const string &parentName) {
auto fnd = childMap.find(parentName);
if (fnd == childMap.end()) {
return nullopt;
}
const Subclasses::Entries children = fnd->second;

Subclasses::Entries out;
out.insert(children.begin(), children.end());
for (const auto &[name, _type] : children) {
auto descendants = Subclasses::descendantsOf(childMap, name);
if (descendants) {
out.insert(descendants->begin(), descendants->end());
}
}

return out;
}

// Manually patch the child map to account for inheritance that happens at runtime `self.included`
// Please do not add to this list.
void Subclasses::patchChildMap(Subclasses::Map &childMap) {
childMap["Opus::SafeMachine"].insert(childMap["Opus::Risk::Model::Mixins::RiskSafeMachine"].begin(),
childMap["Opus::Risk::Model::Mixins::RiskSafeMachine"].end());

childMap["Chalk::SafeMachine"].insert(childMap["Opus::SafeMachine"].begin(), childMap["Opus::SafeMachine"].end());

childMap["Chalk::ODM::Model"].insert(make_pair("Chalk::ODM::Private::Lock", autogen::Definition::Type::Class));
}

vector<string> Subclasses::serializeSubclassMap(const Subclasses::Map &descendantsMap,
const vector<string> &parentNames) {
vector<string> descendantsMapSerialized;

for (const string &parentName : parentNames) {
auto fnd = descendantsMap.find(parentName);
if (fnd == descendantsMap.end()) {
continue;
}
const Subclasses::Entries children = fnd->second;

descendantsMapSerialized.emplace_back(parentName);

vector<string> serializedChildren;
for (const auto &[name, type] : children) {
// Ignore Modules
if (type != autogen::Definition::Type::Class) {
continue;
}
serializedChildren.emplace_back(fmt::format(" {}", name));
}

fast_sort(serializedChildren);
descendantsMapSerialized.insert(descendantsMapSerialized.end(), make_move_iterator(serializedChildren.begin()),
make_move_iterator(serializedChildren.end()));
}

return descendantsMapSerialized;
}

// Generate a list of strings representing the descendants of a given list of parent classes
//
// e.g.
// Parent1
// Child1
// Parent2
// Child2
// Child3
//
// This effectively replaces pay-server's `DescendantsMap` in `InheritedClassesStep` with a much
// faster implementation.
vector<string> Subclasses::genDescendantsMap(Subclasses::Map &childMap, vector<string> &parentNames) {
Subclasses::patchChildMap(childMap);

// Generate descendants for each passed-in superclass
fast_sort(parentNames);
Subclasses::Map descendantsMap;
for (const string &parentName : parentNames) {
// Skip parents that the user asked for but which don't
// exist or are never subclassed.
auto fnd = childMap.find(parentName);
if (fnd == childMap.end()) {
continue;
}

auto descendants = Subclasses::descendantsOf(childMap, parentName);
if (!descendants) {
descendantsMap[parentName];
}

descendantsMap.emplace(parentName, *descendants);
}

return Subclasses::serializeSubclassMap(descendantsMap, parentNames);
};

} // namespace sorbet::autogen
29 changes: 29 additions & 0 deletions main/autogen/subclasses.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
#ifndef AUTOGEN_SUBCLASSES_H
#define AUTOGEN_SUBCLASSES_H
#include "common/common.h"
#include "main/autogen/autogen.h"

namespace sorbet::autogen {

class Subclasses final {
public:
typedef std::pair<std::string, Definition::Type> Entry;
typedef UnorderedSet<Entry> Entries;
typedef UnorderedMap<std::string, Entries> Map;

static std::optional<Subclasses::Map> listAllSubclasses(core::Context ctx, ParsedFile &pf,
const std::vector<std::string> &absoluteIgnorePatterns,
const std::vector<std::string> &relativeIgnorePatterns);
static std::vector<std::string> genDescendantsMap(Subclasses::Map &childMap, std::vector<std::string> &parentNames);

private:
static void patchChildMap(Subclasses::Map &childMap);
static bool isFileIgnored(const std::string &path, const std::vector<std::string> &absoluteIgnorePatterns,
const std::vector<std::string> &relativeIgnorePatterns);
static std::optional<Entries> descendantsOf(const Subclasses::Map &childMap, const std::string &parents);
static std::vector<std::string> serializeSubclassMap(const Subclasses::Map &descendantsMap,
const std::vector<std::string> &parentNames);
};

} // namespace sorbet::autogen
#endif // AUTOGEN_SUBCLASSES_H
41 changes: 37 additions & 4 deletions main/options/options.cc
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,7 @@ const vector<PrintOptions> print_options({
{"autogen", &Printers::Autogen, true},
{"autogen-msgpack", &Printers::AutogenMsgPack, true},
{"autogen-classlist", &Printers::AutogenClasslist, true},
{"autogen-subclasses", &Printers::AutogenSubclasses, true},
{"plugin-generated-code", &Printers::PluginGeneratedCode, true},
});

Expand Down Expand Up @@ -107,6 +108,7 @@ vector<reference_wrapper<PrinterConfig>> Printers::printers() {
Autogen,
AutogenMsgPack,
AutogenClasslist,
AutogenSubclasses,
PluginGeneratedCode,
});
}
Expand Down Expand Up @@ -342,6 +344,13 @@ cxxopts::Options buildOptions() {
cxxopts::value<string>()->default_value(empty.errorUrlBase), "url-base");
// Developer options
options.add_options("dev")("p,print", to_string(all_prints), cxxopts::value<vector<string>>(), "type");
options.add_options("dev")("autogen-subclasses-parent",
"Parent classes for which generate a list of subclasses. "
"This option must be used in conjunction with -p autogen-subclasses",
cxxopts::value<vector<string>>(), "string");
options.add_options("dev")("autogen-subclasses-ignore",
"Like --ignore, but it only affects `-p autogen-subclasses`.",
cxxopts::value<vector<string>>(), "string");
options.add_options("dev")("stop-after", to_string(all_stop_after),
cxxopts::value<string>()->default_value("inferencer"), "phase");
options.add_options("dev")("no-stdlib", "Do not load included rbi files for stdlib");
Expand Down Expand Up @@ -513,7 +522,7 @@ void readOptions(Options &opts, int argc, char *argv[],
if (p.at(0) == '/') {
opts.absoluteIgnorePatterns.emplace_back(pNormalized);
} else {
opts.relativeIgnorePatterns.push_back(fmt::format("/{}", pNormalized));
opts.relativeIgnorePatterns.emplace_back(fmt::format("/{}", pNormalized));
}
}
}
Expand Down Expand Up @@ -600,13 +609,37 @@ void readOptions(Options &opts, int argc, char *argv[],
}
opts.disableWatchman = raw["disable-watchman"].as<bool>();
opts.watchmanPath = raw["watchman-path"].as<string>();
if ((opts.print.Autogen.enabled || opts.print.AutogenMsgPack.enabled || opts.print.AutogenClasslist.enabled) &&

// Certain features only need certain passes
if ((opts.print.Autogen.enabled || opts.print.AutogenMsgPack.enabled || opts.print.AutogenClasslist.enabled ||
opts.print.AutogenSubclasses.enabled) &&
(opts.stopAfterPhase != Phase::NAMER)) {
logger->error("-p autogen{} must also include --stop-after=namer",
opts.print.AutogenMsgPack.enabled ? "-msgpack" : "");
logger->error("-p autogen{-msgpack,-classlist,-subclasses} must also include --stop-after=namer");
throw EarlyReturnWithCode(1);
}

if (raw.count("autogen-subclasses-parent")) {
if (!opts.print.AutogenSubclasses.enabled) {
logger->error("autogen-subclasses-parent must be used with -p autogen-subclasses");
throw EarlyReturnWithCode(1);
}
for (string parentClassName : raw["autogen-subclasses-parent"].as<vector<string>>()) {
opts.autogenSubclassesParents.emplace_back(parentClassName);
}
}

if (raw.count("autogen-subclasses-ignore") > 0) {
auto rawIgnorePatterns = raw["autogen-subclasses-ignore"].as<vector<string>>();
for (auto &p : rawIgnorePatterns) {
string_view pNormalized = stripTrailingSlashes(p);
if (p.at(0) == '/') {
opts.autogenSubclassesAbsoluteIgnorePatterns.emplace_back(pNormalized);
} else {
opts.autogenSubclassesRelativeIgnorePatterns.emplace_back(fmt::format("/{}", pNormalized));
}
}
}

opts.noErrorCount = raw["no-error-count"].as<bool>();
opts.noStdlib = raw["no-stdlib"].as<bool>();
opts.stdoutHUPHack = raw["stdout-hup-hack"].as<bool>();
Expand Down
Loading

0 comments on commit bbbe525

Please sign in to comment.