GEON: Structure-First Decoding

A structural decoding layer for language models. LLMs guess tokens.
GEON enforces structure.

Resolve structure first, then select tokens.

This runs a 10-task code generation benchmark comparing:

• Baseline token decoding
• GEON (structure-first decoding)
• Reproduce the benchmark results locally:

python geon_eval_harness_v2.py

next token ← p(token | context)

structure → valid options → token selection

def is_prime(n):
    if n < 2:

return True  ❌

return False  ✅

def add(a, b):
    return (a + b

return (a + b  ❌

return (a + b)  ✅

We evaluated a small Python code-generation harness on three tasks:

• factorial
• sum_list
• max_element

Both methods produced syntactically valid Python.

The difference appears at the structural and semantic level:

• Baseline LLM often generates plausible but incorrect programs
• GEON restricts generation to structurally admissible continuations

This improves not just syntax, but functional correctness.

We evaluated GEON against a baseline token-selection approach on 10 Python code generation tasks:

• factorial
• sum_list
• max_element
• count_vowels
• reverse_string
• is_even
• is_sorted
• count_positive
• first_char
• square_list

Both methods produce syntactically valid Python.

However:

• The baseline often generates plausible but incorrect programs
• GEON restricts generation to structurally admissible continuations

This leads to consistent semantic correctness across all tasks.

GEON enforces structure before token selection.

Standard decoding selects the next token based on probability: argmax p(token)

GEON changes this process:

1. Tokens are mapped into equivalence classes (ECs)
2. ECs are evaluated under structural constraints (S1 field)
3. Only structurally admissible tokens are allowed
4. Sampling happens within this reduced set

This enforces:

• syntactic closure (e.g. parentheses must match)
• logical consistency (e.g. conditional branches align)
• structural validity before generation

Most LLM errors are not due to lack of knowledge, but lack of structure:

• incomplete code
• invalid syntax
• inconsistent logic

GEON addresses this at the decoding level.

Instead of correcting outputs after generation, it prevents invalid outputs from being produced.

GEON does not ask:

"what token is most likely?"

It asks:

"what tokens are structurally valid?"

Then selects among them.

The name GEON is inspired by the concept of geons introduced by John Archibald Wheeler.

Wheeler used “geons” to describe structures that are defined not by material substance, but by the relationships and constraints that hold them together.

GEON applies a similar idea to generated sequences such as code and structured language:

• meaning is not just in tokens
• it emerges from structure and admissibility
• validity is defined by constraints, not probability

In this sense, GEON treats sequence generation as a structural process rather than a purely probabilistic one.