![Page 1: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/1.jpg)
The Future ofAST-Matcher based Refactoring
Stephen KellyEuroLLVM 2019
steveire.wordpress.com@steveire
![Page 2: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/2.jpg)
Stephen Kelly
@steveire
steveire.wordpress.com
KDE
Qt
CMake
Clang
![Page 3: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/3.jpg)
ASTMatcher-based Refactoring
Scale and Distribute refactoring task Makes intractable problems tractable Allows creating generic reusable tools C++
![Page 4: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/4.jpg)
ASTMatcher-based Refactoring
Learning curve is very steep Hit complexity very fast Requires existing knowledge of Clang APIs Discovery is difficult Multiple domains of input information
AST Nodes, Matchers, Source Locations
Takes lots of slow developer iteration No plugin System C++
![Page 5: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/5.jpg)
Becoming More Novice-Friendly
More documentation More presentations Collaboration New features in existing tools
Workflow Discovery Debugging
New tools Speed
New APIs
![Page 6: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/6.jpg)
Becoming More Novice-Friendly
More documentation More presentations Collaboration New features in existing tools
Workflow Discovery Debugging
New tools Faster iteration
New APIs
![Page 7: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/7.jpg)
Parallel Efforts
ASTER Generate AST Matchers from example code
clang::tooling::Transformation Specify changes based on matched Nodes
Syntax Tree Syntactic Representation and manipulation
![Page 8: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/8.jpg)
Resources and Collaboration
clang-query helps, but not referenced well My vcblog series
3 Part series aimed at Novices
clang-query explorer http://ce.steveire.com/z/pcARNO Upstreaming to godbolt.org
![Page 9: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/9.jpg)
Reduced noise for Novices
Simplified AST to discover top-level Matchers: http://ce.steveire.com/z/sjyYUJ
Detailed AST still available: http://ce.steveire.com/z/OpLliE
Remove ‘invisible’ AST nodes http://ce.steveire.com/z/lHYwEH ignoringImplicit() is not enough http://ce.steveire.com/z/EdnWVg
![Page 10: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/10.jpg)
Workflow (today)
![Page 11: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/11.jpg)
Workflow (future)
![Page 12: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/12.jpg)
Discovery
Close knowledge gap Novice mental model <=> Clang reality
Discover Matchers http://ce.steveire.com/z/lDNQCx
Discover Source Locations http://ce.steveire.com/z/JysGF8
![Page 13: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/13.jpg)
Developer Tooling
Debugger http://ce.steveire.com/z/JgMave
Profiler http://ce.steveire.com/z/wmMd3W
![Page 14: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/14.jpg)
Output independent APIs
Tooling APIs should be output-independent Diagnostics is a good existing example
Output independent AST dump traversal New!
![Page 15: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/15.jpg)
Output independent APIs
Before Now
![Page 16: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/16.jpg)
Output independent APIs
![Page 17: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/17.jpg)
Workflow (future)
![Page 18: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/18.jpg)
Workflow (more-future)
![Page 19: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/19.jpg)
Pending Changes
New Traversal Options Ignore invisible nodes Ignore template instantiations?
Output possible Matchers from clang-query Expose from ast_matchers::dynamic::Registry
Debugger interface for ASTMatchFinder Used for debugging and profiling
AST introspection tool Generate code for source locations etc
![Page 20: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/20.jpg)
class DebuggerInterface {
virtual void DeclareMatcher(
const DynMatcherInterface *Matcher, llvm::StringRef Name,
const DynMatcherInterface *Parent) const = 0;
virtual void CreateBinding(
const DynMatcherInterface *Matcher, llvm::StringRef Name,
const DynMatcherInterface *BindingMatcher) = 0;
virtual void DebugMatch(
const ast_type_traits::DynTypedNode &DynNode,
const DynMatcherInterface *Matcher, bool IsMatch) = 0;
};
![Page 21: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/21.jpg)
clang-ast-introspection
New tool run at build-time Parses clang/AST/AST.h Uses AST-Matchers Generates
C++ API for source location texts JSON data for Javascript bindings
![Page 22: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/22.jpg)
libClangQuery
Library-ify most of clang-query tool struct QueryFactory
{
virtual Query *MakeMatchQuery(
StringRef Source,
const DynTypedMatcher &Matcher)
{ return new MatchQuery(Source, Matcher); }
// etc...
};
![Page 23: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/23.jpg)
ASTMatcher-based Refactoring
Learning curve is very steep Hit complexity very fast Requires existing knowledge of Clang APIs Discovery is difficult Multiple domains of input information
AST Nodes, Matchers, Source Locations
Takes lots of slow developer iteration No plugin System C++
![Page 24: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/24.jpg)
Summary
Mechanical refactoring enabled by Clang Barrier to entry too-great for Clang Novices
Must self-build Clang Reduce verbosity of output by default Add discovery features Reduce domains of data (less AST)
Shorten iteration time Interpreted languages Live result updates
![Page 25: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/25.jpg)
What Now?
Right analysis/Useful work? Is there interest in LLVM? Collaboratorators?
![Page 26: The Future of AST-Matcher based Refactoring · The Future of AST-Matcher based Refactoring Stephen Kelly EuroLLVM 2019 steveire.wordpress.com @steveire. Stephen Kelly ... JSON data](https://reader034.vdocuments.us/reader034/viewer/2022042915/5f515800e5f918157102c2eb/html5/thumbnails/26.jpg)
match questionDecl(
hasAnswer(clearExpr().bind("Answer"))
)
void check(auto const& Result)
{
auto Answer =
Result.Nodes->getAs<ClearExpr>("Answer");
Answer->dump();
}