Recent Commits to lark:master

Add pyparsing porting guide (#1636)

Fix InteractiveParser.copy() sharing the lexer thread with the original (#1641)

Merge pull request #1632 from Sanjays2402/fix/template-search-zero-variable-matches

Fix Template.search() dropping zero-variable matches

Merge pull request #1626 from Sanjays2402/fix/lark-lark-line-continuation

Support backslash line continuations in lark.lark

Merge pull request #1624 from lark-parser/merge-1622-discard-alignment

Merge PR 1622 - fix small collision with previous PR

Tidy Transformer_NonRecursive: move Discard placeholder comment to the filter site

Fix Transformer_NonRecursive corrupting the tree on a non-last Discard

When a callback returned Discard, Transformer_NonRecursive did not push a
result for that node, but a parent still sliced `stack[-len(children):]`.
For a discarded node that is not the last child processed, that slice
over-consumed the stack and pulled an unrelated sibling into the wrong
parent -- e.g. `start[keep, wrap[drop]]` came back as
`start[wrap[keep]]`, silently dropping and re-parenting nodes. The base
Transformer (documented to give the same result) was unaffected.

Push discarded results as placeholders so every node consumes exactly
`len(children)` stack slots, and filter the placeholders when building a
parent's args. A wholly-discarded tree now returns None (this also covers
the root-discard crash) instead of mis-unpacking an empty stack.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: chuenchen309 <48723787+chuenchen309@users.noreply.github.com>

Merge pull request #1623 from amkdown-max/codex/fix-python-comprehension-filters

Fix Python comprehension filters after each for clause

Add template support to Reconstructor (#1613)

Add template support to Reconstructor

Add the full template name from the children matched by the tree
matcher as a meta attribute for the given child so it can be read
when recursing into that child.

Make generated regexps deterministic across processes (#1642)

Make generated regexps deterministic across processes by sorting Pattern.flags

---------

Co-authored-by: Erez Shinan <erezshin@gmail.com>

Merge pull request #1595 from Mirochill/fix-1564-raw-recursive-cache-grammar

Make recursive cache grammar raw

Merge pull request #1637 from MegaIng/update-python-versions

Update Python versions and add a windows run for tests workflow

Update Python versions and add a windows run for tests workflow

Fix Template.search() dropping zero-variable matches

Template.match() returns an empty dict when a template with no variable
placeholders matches a subtree, and None when there is no match. search()
tested the result for truthiness, so the empty dict was treated as "no
match" and every zero-variable match was silently dropped. translate()
therefore never replaced anything for such templates.

search() now tests `res is not None`, matching the contract match()
documents.

Two existing search tests asserted the broken behaviour and are corrected;
added a test that a genuinely different tree still yields nothing, and a
TemplateTranslator test covering a variable-free translation.

Closes #1630

Merge pull request #1605 from nchammas/indenter-docs

Show Indenter property docs + add working links

Support backslash line continuations in lark.lark

The lark.lark meta-grammar did not accept backslash-newline line
continuations, even though grammars parsed by lark support them.
Add a BACKSLASH terminal and ignore it, plus a regression test.

Extract Python comprehension filters rule

Merge pull request #1614 from gaoflow/fix-transformer-nonrecursive-discard-root

Fix Transformer_NonRecursive crashing when root node is discarded

Merge pull request #1619 from apoorvdarshan/fix-1618-lexer-callback-closure

Fix lexer_callbacks silently dropped for keyword terminals (#1618)

Fix Python comprehension filter grammar

Tiny fix

Merge pull request #1604 from nchammas/sphinx-errors

Documentation tweaks to address Sphinx build errors and warnings

Merge pull request #1592 from lark-parser/scan_parse

Add Lark.scan() for finding grammar matches in text, and parsing them.

Fix lexer_callbacks silently dropped for keyword terminals

When two or more terminals each absorb a same-priority keyword (producing
an UnlessCallback) and also have a lexer_callbacks entry, every user
callback except the last silently never fired.

BasicLexer built each CallChain with 'lambda t: t.type == type_', which
closes over the loop variable, so all conditions ended up comparing
against the last terminal's name. Bind type_ per iteration.

Fixes #1618.

scan: Add LineCounter.advance_to(text, pos) for better performance + minor fixes

Scan: Added small warning (possibly TODO if a need arises)

Add more scan() tests

scan(): search start terminals directly so ignored spans aren't skipped

scan(): carry line/column counts across passes instead of recounting

scan: validate config eagerly, reject custom lexers, tighten errors

- Split ParsingFrontend.scan() into _scan() so configuration errors raise on the call
- Lark.scan() now checks parser='lalr' up front too. Reject custom lexer
- Re-raise ConfigurationError instead of counting it as ValueError
- Check source positions on every replayed token, not just the last
docstring.
- Improves docs
- Adds tests

Test that Token end positions survive deepcopy and pickle

Export ScanMatch from the top-level lark package

Token: preserve end_line/end_column/end_pos in __deepcopy__ and __reduce__

Both methods only copied start_pos/line/column, silently dropping the end
positions.

Add Lark.scan() for finding grammar matches in text, and parsing them.

It finds and parses each non-overlapping match, yielding the longest possible match.

Reimplementation of the long-standing #1429 PR on top of the merged
TextSlice support, with some modifications.

Docs: Added recipe for using scan()

Fix Transformer_NonRecursive crashing with ValueError when root is discarded

When the root tree node's callback returns Discard, the stack is left
empty and the unpacking `result, = stack` raises ValueError. The
recursive Transformer.transform() handles this by returning None;
apply the same guard here.

Fixes #1611

show indenter property docs + add working links

tweaks to address build errors

Merge pull request #1603 from sarathfrancis90/fix/declare-rule-name-crash

Raise GrammarError for non-terminal names in %declare

Raise GrammarError for non-terminal names in %declare

%declare is for terminals, but the grammar accepted any symbol, so a
lowercase (rule) name like `%declare foo` defined an abstract rule with
no tree. The compiler later called iter_subtrees() on that missing tree
and crashed with `AttributeError: 'NoneType' object has no attribute
'iter_subtrees'` instead of reporting the bad grammar.

Reject non-terminal names in %declare with a clear GrammarError.

Merge pull request #1601 from lark-parser/fix_issue1598

Fix empty SPPF node from xearley ignore carry-over (fixes #1598)

Fix empty SPPF node from xearley ignore carry-over (fixes #1598)

When an item in to_scan was carried over past an ignored sequence with
item.node=None (e.g. a prediction at ptr=0 that hasn't matched anything
yet), the carry-over still allocated an empty SymbolNode and stored it
in node_cache. That empty node later became the left child of a
PackedNode when the terminal was finally scanned, and crashed
ForestSumVisitor with "max() iterable argument is empty".

Skip SymbolNode creation when item.node is None — there's nothing to
merge, and new_item.node staying None makes the subsequent PackedNode
have left=None, which is well-handled everywhere else.

Merge pull request #1596 from SAY-5/fix-template-in-terminal-error

fix: raise GrammarError for template usage inside terminals

fix: raise GrammarError for template usage inside terminals

Merge pull request #1591 from lark-parser/issue1186

Make Lark generic on transformer return type

Fixes for coverage

Make Lark generic on transformer return type (fixes #1186)

Lark is now Generic[_Return_T]. Two @overload stubs on __init__ bind
_Return_T to the transformer's output type when a transformer= is
passed, or to ParseTree otherwise. parse() return type follows.

Also includes type-only tests.

Test serialized transformer state

Miscellaneous improvements to the behavior and documentation of Visitor, Transformer, Interpreter, and friends (#1543)

* tweak docs

* add missing docstrings, have decorators error when used incorrectly

* add tests for changed behavior

* remove implementation of visit_topdown

* revert changes to _visitor_args_dec

* remove associated tests

* remove more bad tests i added

also tweak new test for decorator use

* factor repeated test

* Small phrasing update

---------

Co-authored-by: Erez Shinan <erezshin+git@gmail.com>

Merge pull request #1586 from Kache/v_args-type-hint

Fix type hint of @v_args() decorator

Fix type hint of @v_args() decorator

Before, the test script below fails Pyright's type-checking:

```
error: "assert_type" mismatch: expected "str" but received "Any | Unknown" (reportAssertTypeFailure)
```

(VSCode's python extension, Pylance, uses Pyright internally)

Test script:

```python
import sys

from lark import Transformer, Token, Tree, v_args

if sys.version_info < (3, 11):
    from typing_extensions import assert_type, reveal_type
else:
    from typing import assert_type, reveal_type

@v_args()
class Foo(Transformer[Token, str]):
    def __init__(self):
        super().__init__()

    def foo(self, meta, args):
        return 'baz'

bar = Foo().transform(Tree('foo', []))

assert_type(bar, str)
```

I've not added this test because it would require adding pyright as a
second typechecker dependency when the project already uses mypy

fix: #1579

Merge pull request #1577 from chanicpanic/issue1576

Bugfix Earley: handle ambiguity from ignored tokens

Bugfix Earley: handle general ambiguity from ignored tokens

Bugfix Earley: handle ambiguity from strings ending in ignored tokens

Merge pull request #1575 from denballakh/patch-1

exceptions.py: sort set of expected things to get deterministic result

exceptions.py: sort set of expected things to get deterministic result

Currently expected tokens are stored in a set, and because of that the order they appear in error message is unstable - different runs may produce different orderings.

Here I sort the set of expected tokens, so that the order is guaranteed to be stable between runs.

Version bump (1.3.1)

Merge pull request #1561 from chanicpanic/sdist-git

Keep sdist in sync with git

Update MANIFEST.in

Merge pull request #1562 from lark-parser/issue1560

Bugfix: Restore support for custom input, alongside text and TextSlice