algebraic threat engine

Experimental Threat Engine

Hi everyone,

I am currently working on a project that I believe is very promising, but it should still be considered experimental at this stage. So far, it is only being used by a handful of people, and their initial suggestions for improvement have already been incorporated into this version.

To test Yara-Analytic under real-world conditions, I am looking for experienced users who are interested in performing in-depth technical analysis. If you have solid expertise in the field and want to test Yara-Analytic, I would appreciate your support.

If you're interested, just send me a short email and I'll send you the link to the tool.

# What is YARA Analytic?

This tool is no ordinary YARA scanner. It is an Algebraic Threat Engine (ATE). My goal was to build a tool that doesn't just blindly find threats but evaluates them mathematically to minimize false positives.

Important: This tool is not for beginners. It requires a solid understanding of how YARA rules, metadata, and pattern matching work.

# Core Features & Technical Architecture

# Freedstorm ATE

Utilizes Max-Aggregation across Severity, Weight, and Score. A hit is only classified as critical if the mathematical relevance reaches the defined threshold.

# Hardened Binary

Compiled with (C++). The scoring logic is protected against tampering. Thanks to Isolation Mode, the tool runs with absolute stability, independent of global Python libraries.

# Deep-Analysis

By providing detailed Hex offsets and a preview of the first 30 bytes of a hit, the engine enables precise forensic evaluation.

# Smart Timeouts

Adjustable timeouts (Default: 5s) prevent "ReDoS"-like YARA rules or manipulated files from stalling the scanning process.

# Filter Policy Logic

The tool implements a complex decision matrix:

• Tag-Filtering: Use --require-tag or --deny-tag to focus on specific threat classes or mark them as "Safe".
• Score-Requirement: With --score-required, only rules providing clean metadata are evaluated.
• Algebra-Threshold: The --min-score N switch defines the red line between "Info" and "Malware".

ATE Calculation Example:

Freedstorm ATE Formula

Score_total = max({ M(Sev), W, S_meta })

• M(Sev): Severity Mapping (Critical=100, High=75, ...)

• W: Weight (Weighting factor from YARA metadata)

• S_meta: Score (Explicit threat value)

# Operation & Lifecycle

yara-analytic --help
usage: yara-analytic [-h] [--update] [--debug] [--test-rules]
                     [--scan-file FILE] [--scan-dir DIR]
                     [--rules-dir PATH] [--verbose] [--silent]
                     [--timeout TIMEOUT] [--min-score N]
                     [--score-required] [--allow-no-score]
                     [--require-tag TAG] [--deny-tag TAG]
                     [--deny-rule NAME]

Freedstorm YARA Analytic v1.6 - Algebraic Threat Engine (ATE)

options:
  -h, --help           Show this help message and exit.
  --update             Update rules (Git required)
  --debug              Generate debug report for troubleshooting
  --test-rules         Display all loaded YARA rules for verification
  --scan-file FILE     Perform deep-analysis scan on a single file
  --scan-dir DIR       Analyze directory recursively
  --rules-dir PATH     Path to YARA rules
                       (Default: /usr/local/share/yara-rules)
  --verbose, -v        Detailed analysis (shows offsets, strings, and tags)
  --silent, -s         Silent mode (suppresses status messages, only hits)
  --timeout TIMEOUT    Scan timeout (N) in seconds (Default: 5)

Algebra & Filter Options:
  --min-score N        Minimum Freedstorm Algebra score (Default: 50)
  --score-required     Only evaluate rules with explicit score metadata
  --allow-no-score     Treat rules without metadata as a risk as well
  --require-tag TAG    Only use rules with this specific tag
  --deny-tag TAG       Always treat rules with this tag as INFO (Safe)
  --deny-rule NAME     Exclude specific rule names from alerts

Lifecycle Management: To prevent the use of outdated engines, every build has a hard End-of-Life (EOL) date. Current versions ensure maximum detection rates.

# Freedstorm Algebra: How the Score is Generated

A conventional scanner immediately screams "Malware!" at every hit. The Freedstorm Algebra, however, acts like a digital judge.

The Calculation Method

During a scan, the engine automatically extracts three critical metadata points from the YARA rules:

• Severity: How dangerous is the rule classified?
• Weight: What is the weighting of the patterns?
• Score: The threat value in the database.

The engine uses Max-Aggregation logic to calculate the actual relevance for the scan.

The Result: Relevance instead of Noise

• Low Score (< 40): Mostly harmless characteristics.
• High Score (> 70): High probability of genuine malicious code.

# Major Update: Version 1.2 & 1.3

Over the last few days, the engine has received a massive development boost. Thanks to feedback from initial testers, critical performance hurdles and rule integrity issues have been eliminated.

Threat-Intelligence Upgrade: Starting with v1.2, the engine automatically integrates the professional ReversingLabs repository. This provides over 2,100 qualified rule files for analysis.

Key Improvements at a Glance:

Feature / Improvement	v1.1	v1.3
Rule Statistics on Load	No	Yes
Detailed Error Logging (Duplicates)	No	Yes
UTF-8 Safe String Preview	No	Yes
Automatic Archive Skip Counter	No	Yes
Integrated --test-rules Mode	No	Yes

# Technical Focus: Stability on Arch Linux

A special focus was placed on fixing the flex scanner error that could occur when processing large path structures on Arch systems. Additionally, the engine now supports External Variables (such as filename), ensuring compatibility with high-end rule sets like those from Florian Roth (Neo23x0).

In addition, v1.3 now proactively warns against "Over-filtering". If your chosen algebra parameters (Score/Tags) are so restrictive that potentially dangerous hits are masked, the tool will issue a corresponding notification.

# Major Update: Version 1.4 & 1.6

With the latest updates, the Freedstorm Engine has evolved from a pure analysis tool into a hardened security application. The focus was on protecting engine integrity and the seamless integration of professional threat intelligence.

Current Development:

Feature / Security Module	v1.3	v1.6
Anti-Tamper (Time Manipulation Protection)	No	Multi-Layer
Full External Variables Support (extension, owner, etc.)	Partial	Full
Automatic Error Handling (Module Incompatibility)	Basic	Advanced
Integrated Debug Report Generator	No	Yes

# Technical Focus: Overcoming Module Hurdles

A breakthrough in v1.5 was the implementation of dynamic variable mapping. Previously, rules accessing external attributes like extension or filename often led to compilation errors. The Freedstorm ATE now provides these globally, allowing even the most complex signatures from the signature-base to function without manual adjustment.

Furthermore, v1.6 no longer delivers cryptic tracebacks on errors, but categorizes issues into understandable messages such as "Module Incompatibility" or "Syntax Error". For forensic analysis, a full report can be generated via --debug, summarizing all system references and error modes for the developer freedstorm.

# Outlook & Testing Phase

I would appreciate any technical feedback.

Note: Binaries are only valid for a few months at a time to prevent the use of outdated experiments.

# Prerequisites

• YARA: Native libraries installed
• Python Dependencies: yara-python
• VCS: git

📥 Download YARA Analytic v1.3 (Linux)

SHA256: 64f23bb76a602ca25a9d1f84ff741474285c09f17d11d46fce73ca34a0038f98