Self-Learning System

Cellium Agent implements a self-learning system based on Bayesian Bandit, enabling the Agent to improve its decision-making over time.

Policy - Bandit - Action Architecture

The system uses a three-layer architecture:

1. Policy Templates - Multiple strategies (default, efficient, aggressive) with different threshold parameters
2. Bayesian Bandit - Uses Thompson Sampling to select the optimal Policy based on historical performance
3. Action Bandit - When multiple actions are candidates, performs tie-breaking using Heuristic Bias

Learning Process

1. Policy Selection - At session start, Bayesian Bandit selects the optimal policy from available templates
2. Threshold Injection - Selected policy parameters are injected into HeuristicEngine and ActionBandit
3. Action Learning - After each round, update Action's Beta distribution based on FeedbackEvaluator's score
4. n-step Return - Accumulate rewards from recent n rounds for delayed feedback support
5. Data Decay - Every 50 sessions, decay old data (factor 0.99) to prevent overfitting

Feedback Evaluation

Uses segmented design, first distinguishing success/failure:

Success Branch

• Base score: 1.0
• Iteration penalty: fewer iterations = higher score
• Token penalty: deduct points if exceeding threshold
• Smoothness bonus: reward for no stagnation

Failure Branch

• Base score: 0.0
• More stagnation iterations = more points deducted
• Error type affects deduction magnitude

Strategy Gene (GEP)

Based on the Strategy Gene research, Cellium Agent implements compact experience representation:

[HARD CONSTRAINTS]
- Task-specific constraints

[CONTROL ACTION]
- Recommended actions for this task type

[AVOID]
- Patterns that led to failures