Document the "%token" directive for Lemon. This directive has been in place

for a while, but was previously undocumented.

FossilOrigin-Name: 36624d3740a8d095eee061bcc5037deabddb88a53444ec1a956a8af7684efa43

4 files changed, 38 insertions, 14 deletions

diff --git a/doc/lemon.html b/doc/lemon.html
index 457004627..324b3f331 100644
--- a/doc/lemon.html
+++ b/doc/lemon.html
@@ -697,6 +697,7 @@ other than that, the order of directives in Lemon is arbitrary.</p>
 <li><tt><a href='#stack_size'>%stack_size</a></tt>
 <li><tt><a href='#start_symbol'>%start_symbol</a></tt>
 <li><tt><a href='#syntax_error'>%syntax_error</a></tt>
+<li><tt><a href='#token'>%token</a></tt>
 <li><tt><a href='#token_class'>%token_class</a></tt>
 <li><tt><a href='#token_destructor'>%token_destructor</a></tt>
 <li><tt><a href='#token_prefix'>%token_prefix</a></tt>
@@ -1079,14 +1080,37 @@ can choose a different start symbol using the
 
 <p>See <a href='#errors'>Error Processing</a>.</p>
 
+<a id='token'></a>
+<h4>4.4.20 The <tt>%token</tt> directive</h4>
+
+<p>Tokens are normally created automatically, the first time they are used.
+Any identifier that begins with an upper-case letter is a token.
+
+<p>Sometimes it is useful to declare tokens in advance, however.  The
+integer values assigned to each token determined by the order in which
+the tokens are seen.  So by declaring tokens in advance, it is possible to
+cause some tokens to have low-numbered values, which might be desirable in
+some grammers, or to have sequential values assigned to a sequence of
+related tokens.  For this reason, the %token directive is provided to
+declare tokens in advance.  The syntax is as follows:
+
+<blockquote>
+<tt>%token</tt> <i>TOKEN</i> <i>TOKEN...</i> <b>.</b>
+</blockquote></p>
+
+<p>The %token directive is followed by zero or more token symbols and
+terminated by a single ".".  Each token named is created if it does not
+already exist.  Tokens are created in order.
+
+
 <a id='token_class'></a>
-<h4>4.4.20 The <tt>%token_class</tt> directive</h4>
+<h4>4.4.21 The <tt>%token_class</tt> directive</h4>
 
 <p>Undocumented.  Appears to be related to the MULTITERMINAL concept.
 <a href='http://sqlite.org/src/fdiff?v1=796930d5fc2036c7&v2=624b24c5dc048e09&sbs=0'>Implementation</a>.</p>
 
 <a id='token_destructor'></a>
-<h4>4.4.21 The <tt>%token_destructor</tt> directive</h4>
+<h4>4.4.22 The <tt>%token_destructor</tt> directive</h4>
 
 <p>The <tt>%destructor</tt> directive assigns a destructor to a non-terminal
 symbol.  (See the description of the
@@ -1102,7 +1126,7 @@ Other than that, the token destructor works just like the non-terminal
 destructors.</p>
 
 <a id='token_prefix'></a>
-<h4>4.4.22 The <tt>%token_prefix</tt> directive</h4>
+<h4>4.4.23 The <tt>%token_prefix</tt> directive</h4>
 
 <p>Lemon generates #defines that assign small integer constants
 to each terminal symbol in the grammar.  If desired, Lemon will
@@ -1129,7 +1153,7 @@ to each of the #defines it generates.</p>
 </pre>
 
 <a id='token_type'></a><a id='ptype'></a>
-<h4>4.4.23 The <tt>%token_type</tt> and <tt>%type</tt> directives</h4>
+<h4>4.4.24 The <tt>%token_type</tt> and <tt>%type</tt> directives</h4>
 
 <p>These directives are used to specify the data types for values
 on the parser's stack associated with terminal and non-terminal
@@ -1166,7 +1190,7 @@ entry parser stack will require 100K of heap space.  If you are willing
 and able to pay that price, fine.  You just need to know.</p>
 
 <a id='pwildcard'></a>
-<h4>4.4.24 The <tt>%wildcard</tt> directive</h4>
+<h4>4.4.25 The <tt>%wildcard</tt> directive</h4>
 
 <p>The <tt>%wildcard</tt> directive is followed by a single token name and a
 period.  This directive specifies that the identified token should
diff --git a/manifest b/manifest
index 10ed18ea2..1a98ce06e 100644
--- a/manifest
+++ b/manifest
@@ -1,5 +1,5 @@
-C For\sthe\ssqlite3_bind_text16\sTCL\sbinding\sused\sfor\stesting,\sensure\sthat\sthere\nare\sat\sleast\s3\sterminating\szeros,\sso\sthat\sthere\swill\salways\sbe\sa\s\\u0000\ncharacter\seven\sif\sthe\soriginal\sbyte\ssequence\sis\san\sodd\snumber\sof\sbytes.
-D 2021-03-27T16:21:34.590
+C Document\sthe\s"%token"\sdirective\sfor\sLemon.\s\sThis\sdirective\shas\sbeen\sin\splace\nfor\sa\swhile,\sbut\swas\spreviously\sundocumented.
+D 2021-03-28T20:44:01.313
 F .fossil-settings/empty-dirs dbb81e8fc0401ac46a1491ab34a7f2c7c0452f2f06b54ebb845d024ca8283ef1
 F .fossil-settings/ignore-glob 35175cdfcf539b2318cb04a9901442804be81cd677d8b889fcc9149c21f239ea
 F LICENSE.md df5091916dbb40e6e9686186587125e1b2ff51f022cc334e886c19a0e9982724
@@ -38,7 +38,7 @@ F configure c658869fc056a2460a2212fb2960e410cd24e3fb9c55640fbc78d15b48810936 x
 F configure.ac 4e4b58b32f88c8da9914a2f2c3158f80e69907eccc019fcc7e3ba14ffd91c640
 F contrib/sqlitecon.tcl 210a913ad63f9f991070821e599d600bd913e0ad
 F doc/F2FS.txt c1d4a0ae9711cfe0e1d8b019d154f1c29e0d3abfe820787ba1e9ed7691160fcd
-F doc/lemon.html 1bb72ece6271df0d901d233551dd985f2c6ba30d09382cf2d321ed951ab57491
+F doc/lemon.html efc0cd2345d66905505d98f862e1c571512def0ceb5b016cb658fd4918eb76a3
 F doc/pager-invariants.txt 27fed9a70ddad2088750c4a2b493b63853da2710
 F doc/trusted-schema.md 33625008620e879c7bcfbbfa079587612c434fa094d338b08242288d358c3e8a
 F doc/vdbesort-memory.md 4da2639c14cd24a31e0af694b1a8dd37eaf277aff3867e9a8cc14046bc49df56
@@ -1830,7 +1830,7 @@ F tool/genfkey.test b6afd7b825d797a1e1274f519ab5695373552ecad5cd373530c63533638a
 F tool/getlock.c f4c39b651370156cae979501a7b156bdba50e7ce
 F tool/index_usage.c f62a0c701b2c7ff2f3e21d206f093c123f222dbf07136a10ffd1ca15a5c706c5
 F tool/kvtest-speed.sh 4761a9c4b3530907562314d7757995787f7aef8f
-F tool/lemon.c d44ba4f03427c9bd34b601f315fe77c2b6d4bd215801a0259aeedbcc4c94a95c
+F tool/lemon.c a5acddd3eec6a388872aae6efc7563336348a9c45e5563642f77e8e3a50e859d
 F tool/lempar.c 1d3d075da18681c67ecc66c1f171e7094e18cd2cfba6a8a1bd4f3f639d6656e1
 F tool/libvers.c caafc3b689638a1d88d44bc5f526c2278760d9b9
 F tool/loadfts.c c3c64e4d5e90e8ba41159232c2189dba4be7b862
@@ -1911,7 +1911,7 @@ F vsixtest/vsixtest.tcl 6a9a6ab600c25a91a7acc6293828957a386a8a93
 F vsixtest/vsixtest.vcxproj.data 2ed517e100c66dc455b492e1a33350c1b20fbcdc
 F vsixtest/vsixtest.vcxproj.filters 37e51ffedcdb064aad6ff33b6148725226cd608e
 F vsixtest/vsixtest_TemporaryKey.pfx e5b1b036facdb453873e7084e1cae9102ccc67a0
-P f472fd64434cb2eac169587faa34bec489bc82002d529bc72dbfbba5251e5517
-R 9e091a36f447c6ad777be9227680f2e9
+P c23d092f37c535f88a47e9f55aa7b2fd40cfd5c50eec9191a68073d7f9141ecd
+R 1962b187e50d8ed528041ea3add6901a
 U drh
-Z bdab8b445f301b8530dd8bb9c8ad1843
+Z 89ccc7976bdb2c2ae9dddc0aacedcb7d
diff --git a/manifest.uuid b/manifest.uuid
index b74299ff5..1a5f64699 100644
--- a/manifest.uuid
+++ b/manifest.uuid
@@ -1 +1 @@
-c23d092f37c535f88a47e9f55aa7b2fd40cfd5c50eec9191a68073d7f9141ecd
\ No newline at end of file
+36624d3740a8d095eee061bcc5037deabddb88a53444ec1a956a8af7684efa43
\ No newline at end of file
diff --git a/tool/lemon.c b/tool/lemon.c
index 06ba1be2b..75fc7aa2f 100644
--- a/tool/lemon.c
+++ b/tool/lemon.c
@@ -2708,7 +2708,7 @@ static void parseonetoken(struct pstate *psp)
       ** in order to control their assigned integer number.  The number for
       ** each token is assigned when it is first seen.  So by including
       **
-      **     %token ONE TWO THREE
+      **     %token ONE TWO THREE.
       **
       ** early in the grammar file, that assigns small consecutive values
       ** to each of the tokens ONE TWO and THREE.