Attribute grammars Course "Software Language Engineering" University of Koblenz-Landau Department of Computer Science Ralf Lämmel Software Languages Team © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 1 Overview * Parse tree + 4 2 3 20 * Attributed parse tree 2 5 + 2 3 4 4 3 © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 2 Motivating example: Semantic analyses for a DSL © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 3 Semantic analyses for DSL for finite state machines !"#$%#&'"()*#*")+(#($,-$.'#-(/-0/"-*1( state S1; state S2; hablekstate = succ k "S3; reachable j !"# ! s #succ trans S1 -> S2; hable1trans = {S2 } " S2 reachable -> S1; 2 -> S3; hable2trans = {S1,S3 }S2 " reachable 1 " reachable3 j hable3 = # !%# !$# k 2345(365(378( [http://dx.doi.org/10.1007/978-3-642-18023-1_4 and accompanying slides, “An Introductory Tutorial on JastAdd Attribute Grammars” by Görel Hedin] "(&1(;<"=>/0,?*(,*"-#@0?A( 2345(368( 2345(378( 2365(378( © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle (;?=()0'.@0?(,C(:#'.")($#?(&"( 4 •!1.%&('23&4(35.#"6*1.%&('23&4-%.7"-* •!8-%-"4-.%&('23&(* •!8-%-"4."%#$%,9"* •!!%'&*:.37.%)* •!@"A"."&#"*%B.',5-"(* •!C399"#23&*%B.',5-"(* •!C'.#59%.*%B.',5-"(* •!D%E%*#3F"* DSL support for name resolution / analysis !"#"$&#'()*$ transition links* !"#"$% 8; !"#"$% 8< +,#*-)./*% 8;*>?*8< +,#*-)./*% 8<*>?*8; !"#"$% 8= +,#*-)./*% 8<*>?*8= source* target* S2.reachable = {S1, S2, S3} Resolve source/target links and transitions. [http://dx.doi.org/10.1007/978-3-642-18023-1_4 and accompanying slides, “An Introductory Tutorial on JastAdd Attribute Grammars” by Görel Hedin] (C) 2009 Görel Hedin This slide was taken from G. Hedin’s GTT © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 5 DSL support for reachability analysis !"#$%&'()*)+,#,')-#./01')2#134#3') state state state trans trans trans S1; S2; state S1; state S2; S3; trans S1 -> S2; trans S2 -> S1; state S3; S1 -> S2; trans S2 -> S3; S2 -> S1; S2 -> S3; !"#$%#&'(')*+ 5+67)+87)+9:) !"# 5):) !%# !$# 5+67)+87)+9:) [http://dx.doi.org/10.1007/978-3-642-18023-1_4 and accompanying slides, “An Introductory Tutorial on JastAdd Attribute Grammars” by Görel Hedin] © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle (C) 2009 Görel Hed This slide was taken from G. Hedin’s 6 G Overview Attribute grammar = context-free grammar + attribute declarations per nonterminals + semantic rules for relating attributes in derivation tree Applications Semantic analysis (e.g., name analysis) Language translation (as in compilation) © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 7 A simple attribute grammar example due to D. Knuth. © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 8 Binary to decimal conversion 0 ➜ 0 1 ➜ 1 10 ➜ 2 11 ➜ 3 100 ➜ 4 101 ➜ 5 101.01 ➜ 5 . 25 © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 9 Context-free grammar of binary numbers (EBNF) number = bit+ (“.” bit+)? bit = "0" | "1" © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 10 Context-free grammar of binary numbers (BNF) number ::= bits rest bits ::= bit bits ::= bit bits bit ::= "0" bit ::= "1" rest ::= ε rest ::= "." bits © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 11 Binary-to-decimal conversion Attribute numbers, bit sequences, bits with decimal values. Use extra helper attributes for bit position and length. 5.25 0 ➜ 0 1 ➜ 1 10 ➜ 2 11 ➜ 3 100 ➜ 4 101 ➜ 5 10 1 .01 ➜ 5 . 25 4 bit 1 0 bit 0 number 5 bits 1 bits 1/4 bits 1 bits 0 bits 1 bit 1 rest 1/4 0 . bit 0 © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 1/4 bit 1 12 Attributes for binary-to-decimal conversion Nonterminal Attribute Category Type number Val syn float bits Pos inh integer bits Len syn natural bits Val syn float bit Pos inh integer bit Val syn float rest Val syn float © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 13 Inherited vs. synthesized attributes Intuitions Inherited attributes are passed down the tree. Synthesized attributes are passed up the tree. Consider attributes per rule X0 ::= X1 ... Xn: Defined attributes: SYN(X0), INH(X1), ..., INH(Xn) Applied attributes: INH(X0), SYN(X1), ..., SYN(Xn) There must be semantic rules for each production such that defined attributes are actually “defined” in terms of applied ones. © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 14 Semantic rules per production © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 15 Semantic rules per production number ::= bits rest bits.Pos = bits.Len - 1 number.Val = bits.Val + rest.Val © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 16 Semantic rules per production bit ::= "0" bit ::= "1" bit.Val = 0 bit.Val = 2 ^ bit.Pos © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 17 Semantic rules per production bits ::= bit bits0 ::= bit bits1 bit.Pos = bits.Pos bits.Val = bit.Val bits.Len = 1 bit.Pos = bits0.Pos bits1.Pos = bits0.Pos - 1 bits0.Val = bit.Val + bits1.Val bits0.Len = bits1.Len + 1 © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 18 Semantic rules per production rest ::= ε rest.Val = 0 rest ::= "." bits rest.Val = bits.Val bits.Pos = -1 © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 19 Attribute evaluation © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 20 Equational system Assume that inherited attributes of start symbol are defined. Instantiate semantic rules with node identities for nonterminal nodes. Solve the equational system by substitution. An attribute grammar is well-formed if the system has a unique solution. (In particular, cyclic dependencies must not be present. © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 21 Equational system b.Pos = b.Len - 1 a.Val = b.Val + c.Val a:number b:bits c:rest d:bits e:bits f:bits g:bits k:bit l:bit 0 1 h:bit i:bit j:bit 1 0 1 . h.Pos = b.Pos d.Pos = b.Pos - 1 b.Val = h.Val + d.Val b.Len = d.Len + 1 ... © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 22 More on evaluation Use work-list algorithm to defer rules until applicable. Special cases: S-attributed: bottom-up evaluation L-attributed: evaluation proceeds from left to right. One pass: defined attributes are computable from applied ones in some order. Multi pass: a statically known sequence of passes can be used. © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 23 An S-attributed variation number ::= bits rest bits ::= bit bits0 ::= bits1 bit bit ::= "0" bit ::= "1" rest ::= ε rest ::= "." bits number.Val = bits.Val + rest.Val bits.Val = bit.Val bits0.Val = 2*bits1.Val+bit.Val bit.Val = 0 bit.Val = 1 rest.Val = 0 rest.Val = bits.Val / 2 ^ bits.Len bits.Len = 1 bits0.Len = bits1.Len + 1 © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 24 Implementation of attribute grammars © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 25 Overall options Use designated attribute grammar system Yacc (?), ANTLR (?), Eli, JastAdd, ... Use expressive, declarative language Prolog with constraints, Haskell with laziness Implement in free-wheeling code ANTLR/Java hybrid, Recursive descent © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 26 Binary-to-decimal number conversion with ANTLR https://github.com/slecourse/slecourse/tree/master/sources/attributeGrammars/knuth/antlr © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 27 Mapping attribute grammars to Haskell with laziness © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 28 Mapping attribute grammars to Haskell with laziness Nonterminals become functions. One equation per production. Attributes become function arguments and results -- tuples thereof. Inherited attributes make up the argument tuple. Synthesized attributes make up the result tuple. © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 29 Algebraic datatypes instead of BNF -- Non-negative binary numbers with optional binary places data Number = Number Bits Rest -- Non-empty bit sequences data Bits = OneBit Bit | MoreBits Bit Bits -- Single bits data Bit = Zero | One -- Binary places, if any data Rest = None | Places Bits © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle number bits bits bit bit rest rest ::= bits rest ::= bit ::= bit bits ::= "0" ::= "1" ::= ε ::= "." bits 30 One function per nonterminal evalNumber :: Number -> Float evalNumber (Number bits rest) = val0 where (len1, val1) = evalBits bits pos1 pos1 = len1 - 1 val2 = evalRest rest val0 = val1 + val2 number ::= bits rest bits.Pos = bits.Len - 1 number.Val = bits.Val + rest.Val © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 31 One function per nonterminal bit ::= "0" evalBit :: Bit -> Int -> Float evalBit Zero _pos = 0 evalBit One pos = 2 ^^ pos bit.Val = 0 bit ::= "1" bit.Val = 2 ^ bit.Pos © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 32 One function per nonterminal evalBits :: Bits -> Int -> (Int, Float) evalBits (OneBit bit) pos = (1, evalBit bit pos) evalBits (MoreBits bit bits) pos0 = (len0, val0) where val1 = evalBit bit pos0 (len1, val2) = evalBits bits pos1 pos1 = pos0 - 1 len0 = len1 + 1 val0 = val1 + val2 bits ::= bit bit.Pos = bits.Pos bits.Val = bit.Val bits.Len = 1 bits0 ::= bit bits1 bit.Pos = bits0.Pos bits1.Pos = bits0.Pos - 1 bits0.Val = bit.Val + bits1.Val bits0.Len = bits1.Len + 1 © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 33 One function per nonterminal evalRest :: Rest -> Float evalRest None = 0 evalRest (Places bits) = val where (_len, val) = evalBits bits pos pos = -1 rest ::= ε rest.Val = 0 rest ::= "." bits © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle rest.Val = bits.Val bits.Pos = -1 34 Mapping attribute grammars to Prolog with constraints © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 35 Mapping attribute grammars to Prolog with constraints Nonterminals become predicate. One clause per production. Attributes become logical variables. One position per attribute declaration. Use logical variables to cover more than L-attributed grammars. Use constraint-logic programming to cover more AGs. © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 36 A definite clause grammar (DCG) for BNF number --> bits, rest. bits --> bit. bits --> bit, bits. bit --> ['0']. bit --> ['1']. rest --> []. rest --> ['.'], bits. Grammar-aware formalism related Definite Clause Programs of (Prolog) which can be translated to Prolog. © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 37 The corresponding Prolog program obtained by expansion of the DCG number(A, C) :- bits(A, B), rest(B, C). bits(A, B) :- bit(A, B). bits(A, C) :- bit(A, B), bits(B, C). rest(A, A). rest(['.'|A], B) :- bits(A, B). bit(['0'|A], A). bit(['1'|A], A). Thus, an accumulator deals with parsing. 1st arg for initial input; 2nd arg for remaining input. © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 38 One predicate per nonterminal number(Val0) --> { Pos1 is Len1 - 1 }, bits(Pos1, Len1, Val1), rest(Val2), { Val0 is Val1 + Val2 }. ERROR: is/2: Arguments are not sufficiently instantiated number ::= bits rest bits.Pos = bits.Len - 1 number.Val = bits.Val + rest.Val Computation over attributes © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 39 One predicate per nonterminal number(Val0) --> bits(Pos1, Len1, Val1), { Pos1 is Len1 - 1 }, rest(Val2), { Val0 is Val1 + Val2 }. Same error as before while trying “2 ^ bit.Pos” number ::= bits rest bits.Pos = bits.Len - 1 number.Val = bits.Val + rest.Val © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 40 One predicate per nonterminal (Verwendung von CLP(R)) number(Val0) --> bits(Pos1, Len1, Val1), rest(Val2), { { Pos1 =:= Len1 - 1 } }, Computation over attributes number ::= bits rest bits.Pos = bits.Len - 1 number.Val = bits.Val + rest.Val { { Val0 =:= Val1 + Val2 } }. Constraint form © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 41 One predicate per nonterminal (Leverage CLP(R)) bit(_Pos, Val) --> ['0'], { { Val =:= 0 } }. bit(Pos, Val) --> ['1'], { { Val =:= 2 ^ Pos } }. bit ::= "0" bit.Val = 0 bit ::= "1" bit.Val = 2 ^ bit.Pos © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 42 One predicate per nonterminal (Leverage CLP(R)) bits(Pos, Len, Val) --> bit(Pos, Val), { { Len =:= 1 } }. bits(Pos0, Len0, Val0) --> bit(Pos0, Val1), bits(Pos1, Len1, Val2), { { Pos1 =:= Pos0 - 1 } }, { { Len0 =:= Len1 + 1 } }, { { Val0 =:= Val1 + Val2 } }. bits ::= bit bit.Pos = bits.Pos bits.Val = bit.Val bits.Len = 1 bits0 ::= bit bits1 bit.Pos = bits0.Pos bits1.Pos = bits0.Pos - 1 bits0.Val = bit.Val + bits1.Val bits0.Len = bits1.Len + 1 © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 43 One predicate per nonterminal (Leverage CLP(R)) rest(Val) --> { { Val =:= 0 } }. rest(Val) --> ['.'], bits(Pos, _Len, Val), { { Pos =:= -1 } }. rest ::= ε rest.Val = 0 rest ::= "." bits © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle rest.Val = bits.Val bits.Pos = -1 44 structors Leaf and Bin by their corresponding operations from the algebra a that is passed as an argument.1 We now want to construct a function rep min :: Tree -> Tree that retur a Tree with the same “shape” as its argument Tree, but with the values in leaves replaced by the minimal value occurring in the original tree. In figure an example of an argument with its result is given. The RepMin problem Bin Bin Bin Leaf 3 Leaf 4 Bin Leaf 3 Leaf 5 Leaf 3 Leaf 3 Fig. 1. The function rep min [http://dx.doi.org/10.1007/10704973_4, “Designing and Implementing Combinator Languages” by S. Doaitse Swierstra, Pablo R. Azero Alcocer, João Saraiva] 1 Note that this function could have been defined using the language PolyP from t © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 45 Straightforward Solution The straightforward solution to the Rep Min problem consists of a function in which cata Tree is called twice: once for computing the minimal leaf value, and once for constructing the resulting Tree. The function replace min that solves the problem in this way is given in listing 2. Notice that the variable m is used as a global variable in the rep algebra, that in its turn is an argument to the tree constructing call of cata Tree. In figure 2 we have shown the flow of the data in a recursive call of cata Tree, when computing the minimal value. One of the disadvantages of this solution is that, since Computing the minimum Bin 3 min Leaf Bin 3 3 4 min Leaf 4 4 Leaf 5 5 Fig. 2. Computing the minimum value [http://dx.doi.org/10.1007/10704973_4, “Designing and Implementing Combinator Languages” by S. Doaitse Swierstra, Pablo R. Azero Alcocer, João Saraiva] © 2012-15 Ralftwice, Lämmel, Software http://softlang.wikidot.com/course:sle Tree in the Language courseEngineering of the computation the pattern matching we call cata 46 Building the result Designing and Implementing Combinator Languages 157 Bin Bin Leaf Bin Leaf Bin 3 Leaf m Leaf Leaf Leaf 4 5 [http://dx.doi.org/10.1007/10704973_4, “Designing and Implementing Combinator Languages” by S. Doaitse Swierstra, Pablo R. Azero Alcocer, João Saraiva] Fig. 3. The flow of information when building the result © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 47 The abstract syntax of tree [Leaf] Tree ::= Int [Bin] Tree ::= Tree Tree © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 48 An attribute grammar for RepMin [Start] RepMin ::= Tree RepMin.result = Tree.result Tree.new = Tree.min An extra start symbol to distribute global minimum for replacement [Leaf] Tree ::= Int Tree.result = makeLeaf(Tree.new) Tree.min = Int.val [Bin] Tree0 ::= Tree1 Tree2 Tree0.min = min(Tree1.min, Tree2.min) Tree1.new = Tree0.new Tree2.new = Tree0.new Tree0.result = makeBin(Tree1.result, Tree2.result) © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 49 The two cases for building the result 4 5 (Think of it as a visualized attribute grammar!) Leaf Leaf Fig. 3. The flow of information when building the result m Leaf Bin m Bin Leaf _ m lfun lt m rfun rt Fig. 4. The building blocks [http://dx.doi.org/10.1007/10704973_4, “Designing and Implementing Combinator Languages” by S. Doaitse Swierstra, Pablo R. Azero Alcocer, João Saraiva] m Bin © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 50 The RepMin problem in Haskell © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 51 Abstract syntax in Haskell data Tree = Leaf Int | Bin Tree Tree Bin (Leaf 3 ) (Bin (Lea f 4) (Leaf 5)) © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 52 Use of two traversals (This is not a direct mapping of the attribute grammar.) module RepMin where import Tree minTrav :: Tree -> Int minTrav (Leaf x) = x minTrav (Bin l r) = min (minTrav l) (minTrav r) repTrav :: Int -> Tree -> Tree repTrav x (Leaf _) = Leaf x repTrav x (Bin l r) = Bin (repTrav x l) (repTrav x r) repMin :: Tree -> Tree Challenges 1) Can we define an attribute grammar instead? 2) Can we use the attribute grammar to perform just one traversal? repMin t = t' where m = minTrav t t' = repTrav m t © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 53 Direct mapping of the AG to Haskell: module RepMin where one traversal (while requiring laziness) import Tree repMinTrav :: Tree -> Int -> (Int, Tree) repMinTrav (Leaf x) new = (x, Leaf new) repMinTrav (Bin l r) new = (min ml mr, Bin tl tr) where (ml, tl) = repMinTrav l new (mr, tr) = repMinTrav r new data Tree = Leaf Int | Bin Tree Tree repMin :: Tree -> Tree repMin t = t' where (m, t') = repMinTrav t m © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 54 Mapping attribute grammars to Prolog with unification © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 55 Mapping rules for Prolog One parameter position for input pattern. One parameter position for tree-typed result. Use logical variables for unknown attributes. Use unification to propagate eventual attribute values. No CLP needed because no operations needed on unknowns. © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 56 RepMin in Prolog repMin(T1,T2) :tree(New,T1,Min,T2), New = Min. New and Min could be just one variable. We defer unification just for clarity. [Start] tree(New,leaf(Min),Min,leaf(New)). tree(New,bin(L1,R1),Min,bin(L2,R2)) :tree(New,L1,MinL,L2), tree(New,R1,MinR,R2), Min is min(MinL,MinR). RepMin ::= Tree RepMin.result = Tree.result Tree.new = Tree.min [Leaf] Tree ::= Int Tree.result = makeLeaf(Tree.new) Tree.min = Int.val [Bin] Tree0 ::= Tree1 Tree2 Tree0.min = min(Tree1.min, Tree2.min) Tree1.new = Tree0.new Tree2.new = Tree0.new Tree0.result = makeBin(Tree1.result, Tree2.result) © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 57 Concluding remarks Attribute grammars support language implementation: analyses, translations, … By default, they are highly declarative. They can be implemented in declarative languages. The basic formalism lacks some expressiveness. One might want to look at an advanced system (JastAdd). © 2012-15 Ralf Lämmel, Software Language Engineering http://softlang.wikidot.com/course:sle 58