WIP-0006: Concrete syntax, editor tooling, and teaching profile

Summary

Wheeler has formatting-independent, unambiguous syntax with stable source locations. The compiler lexer records the line, column, and offset of each token. A matching Tree-sitter grammar gives editors named nodes, highlighting, folds, comments, and .w file support without exposing compiler classes.

The language also follows a clear teaching order. It starts with ordinary state and effect barriers, then adds reversible methods and reverse blocks. Later steps cover coherent permutations, unitary circuits and adjoints, measurement, and hybrid target workflows.

Each semantic boundary has a stable syntax node. Editor tools must not infer meaning or merge distinct operations just to simplify highlighting. Examples introduce one boundary at a time, and they never describe measurement as a normal reversible assignment.

Motivation

Wheeler source must work in compilers, editors, notebooks, code browsers, classrooms, and research tools. A parser that depends on line breaks will be fragile in each setting. A parser that accepts broad syntax and discards unsupported nodes is just as harmful.

Quantum programming already asks readers to learn difficult ideas. The syntax should make important boundaries visible without extra punctuation or provider-specific terms.

Editor parsers need a stable concrete tree while type checking and target support continue to grow. Learners need runnable examples, deterministic simulators, useful errors, and exact words for inverse, uncompute, rewind, replay, and retry.

Use cases

• An editor incrementally reparses a reverse block after one keystroke and retains surrounding method nodes.
• A notebook highlights quantum resources, unitary methods, measurements, and reversible modifiers.
• A compiler diagnostic reports the exact line and column of an unresolved state or malformed gate.
• A course begins with Counter, then uses one XOR method both classically and coherently, then introduces Bell/QFT and measurement.
• A syntax-tree consumer can index declarations and calls without loading the compiler or a quantum provider.

Goals

• Make whitespace and line wrapping non-semantic outside tokens and comments.
• Keep delimiters, keywords, and declaration shapes deterministic and easy to recover.
• Provide a maintained Tree-sitter grammar, corpus, highlight queries, and fold queries.
• Keep compiler and Tree-sitter examples synchronized in CI.
• Provide source line, column, and offset from lexical analysis onward.
• Organize executable examples into a documented teaching path.
• Use ASCII source for required syntax while permitting Unicode in comments and future identifiers deliberately.
• Make every conceptual transition explicit enough to explain and inspect.

Non-goals

• Put type checking or target capability logic in Tree-sitter queries.
• Require one editor or language-server protocol implementation.
• Minimize character count at the cost of semantic clarity.
• Introduce symbolic quantum notation as required syntax.
• Teach provider APIs before portable Wheeler semantics.

Terms and semantic model

The concrete syntax tree records source structure, delimiters, and recovery nodes. The semantic model resolves names, effects, ownership, inverse relationships, and target requirements. Tooling may rely on stable named syntax nodes but not on compiler-private Java classes.

A teaching level is a set of already implemented language features and examples, not a weaker execution mode. Code learned at an earlier level remains valid later.

Ownership and boundaries

wheeler-compiler owns the authoritative lexical and semantic diagnostics used for compilation. tree-sitter-wheeler owns incremental concrete parsing and editor queries. WIP-0005 owns language meaning. Documentation and examples own the teaching sequence.

The two parsers share corpus source fixtures and a checked grammar contract, not generated implementation code. WIP-0016 owns the one fixed formatter style, lossless trivia boundary, and mandatory /// file/function documentation policy. This WIP supplies the syntax and recovery contract it must not fork.

Design

Lexical contract

Comments are // or /* ... */. Identifiers use the documented ASCII profile initially. Numeric forms are explicit. Keywords are closed and provider-independent. Operators use longest-match tokenization. Tokens retain line, column, and source offset.

Whitespace, comments, and line breaks may occur between tokens. A statement doesn't need to occupy one line. Semicolons terminate simple statements. Braces delimit classes, methods, and reverse blocks.

Concrete tree

Named nodes include class, computation domain, member declaration, state declaration, qreg declaration, method declaration, modifiers, block, assignment, assertion, call, coherent application, reverse statement, call expression, qubit reference, numeric literal, identifier, and comments.

The grammar avoids semantic ambiguity. Whether a method call is classical, unitary, or invalid is resolved after parsing from declarations and effects.

Error recovery

Class, method, and statement delimiters provide synchronization points. Compiler diagnostics fail closed and identify an expected token plus actual token location. Incremental tooling may retain ERROR nodes while users type.

Teaching sequence

1. Classical state covers fields, arithmetic, assertions, and deterministic execution.
2. Reversible methods introduce generated inverses, reverse calls and blocks, and the difference between history and inverse execution.
3. Coherent values use an exact XOR permutation on a classical bit and a quantum basis state.
4. Unitary circuits add H, controlled operations, QFT, and generated adjoints.
5. Measurement introduces affine consumption, classical observations, and sampling.
6. Hybrid workflows cover parameter binding, jobs, replay, and target capabilities.
7. Advanced systems add dynamic correction, resource estimates, proofs, and future logical hardware.

Every level has at least one checked-in source file and an automated expected result.

Reversibility and history

Syntax and teaching material consistently distinguish reverse from debugger rewind. q.apply(method) describes coherent lifting. Unitary method reversal describes an adjoint. Measurement examples explain replay/retry instead of inverse collapse.

Concurrency and determinism

Parsing is deterministic. The first teaching levels avoid concurrency. Draft WIP-0032 owns later IoScope, request, operation, batch, graph, selection, asynchronous, and required-concurrent syntax. Tooling must not copy Java threads or expose a backend poll state machine as source syntax. Similar punctuation does not make the models equivalent.

Quantum and proof implications

Gate names are ordinary ASCII identifiers and qreg indexing is Java-shaped. Dirac notation may be added as optional sugar only with a canonical ASCII equivalent and tooling coverage. WIP-0011 proof syntax exposes contracts, theorem statements, propositions, proof terms, and experiment declarations as stable nodes. Free-form justification text is not proof evidence.

Bytecode, persistence, and compatibility

Concrete syntax is not bytecode. Syntax changes follow WIP migration rules and Tree-sitter corpus updates. Debug sections may retain source spans but canonical execution does not require source files.

Safety, limits, and failures

The compiler bounds input bytes and characters, token and line counts, token width, declarations, and structured-block nesting before lowering. Block comments must close. Identifiers use the required ASCII profile. Numeric overflow and malformed encodings produce diagnostics instead of partial artifacts. Tree-sitter editor hosts remain responsible for document-size policy. The repository gate applies the grammar to every bounded checked-in source.

Migration and deletion

1. Replace line-oriented parsing with a token and recursive-descent parser.
2. Add tree-sitter-wheeler with corpus and editor queries.
3. Test multiline and compact formatting through both parsers.
4. Publish the concrete syntax and teaching sequence in reference documentation.
5. Delete temporary syntax and stale tutorial scaffolding.

Progress

• [x] Compiler lexer records line, column, and offset and exposes one authoritative lossless token/trivia/comment range stream with exact reconstruction. Syntax-node attachment and formatting remain WIP-0016.
• [x] Compiler parsing is formatting-independent.
• [x] The Wheeler-native compiler accepts one canonical dotted module header and emits the same qualified entry/helper and unqualified theorem identities as stage 0. Malformed headers fail before publication. Imports and multi-file linking remain part of the wider self-hosting module work.
• [x] Tree-sitter grammar, corpus, highlights, and folds exist.
• [x] The Gradle/CI gate runs both compiler and Tree-sitter corpus tests.
• [x] The implemented syntax and teaching path are published in reference documentation.
• [ ] Every teaching level has an executable example.

Testing and acceptance

• [x] Compiler tests parse compact, multiline, and comment-heavy programs.
• [x] Lexer tests cover longest-match operators, comments, numeric forms, locations, ASCII identifiers, and hard source/token limits. Parser tests cover the structured nesting ceiling.
• [x] Tree-sitter generates without conflicts and its initial corpus passes.
• [x] Every checked-in example parses without Tree-sitter ERROR nodes, compiles, and executes.
• [x] Highlight queries compile against the grammar. Generated node types cover fold nodes.
• [x] Malformed delimiter fixtures recover in Tree-sitter and fail with source locations in the compiler.
• [x] The documentation teaches the currently implemented semantic boundaries in dependency order.

Alternatives

Continue line-oriented parsing

Rejected. It makes formatting semantic and produces poor editor recovery.

Use the Java grammar unchanged

Rejected. It would parse many constructs Wheeler cannot type or execute and would hide Wheeler's computation domains and affine resource semantics.

Use symbolic quantum notation as the primary grammar

Rejected. ASCII Java-shaped syntax is easier to type, teach, search, and support across tools. Mathematical notation can remain documentation or future optional sugar.

Open questions

• When should the initial ASCII identifier profile expand to full Unicode identifier classes (owner: language and tooling maintainers. Decision point: before module/package syntax)?
• Which stable syntax-node compatibility policy begins with the first external editor integration (owner: tooling maintainers. Decision point: before grammar package release)?

Integration with reversible concurrency

Structured-task syntax boundary

Tree-sitter may add task-scope, spawn, join, atomic, and witness nodes only after WIP-0005 accepts source spelling. Tooling distinguishes TaskScope from WIP-0032 IoScope and assigns no Java or JavaScript meaning to async, concurrent, parallel, or await.

Teaching introduces private tasks before atomics, atomics before bakery, and VM rewind before witnessed source inverse.

References

• WIP-0039
• WIP-0040

• WIP-0005
• WIP-0011
• WIP-0016
• WIP-0017
• WIP-0032
• Language profile
• tree-sitter-wheeler