Finds.

Procedural Graphs: Self-Evolving Execution Structures for LLM Agents

This paper introduces the Procedural Graph, a framework that organizes procedural knowledge into (procedure, relation, procedure) triplets to guide LLM agents.

ProblemLLM agents often lose track of objectives, invoke tools out of order, and repeat unproductive actions as their interaction history accumulates, due to the lack of structured procedural knowledge.

Use it forImproving long-horizon planning consistency in LLM agents; Preventing tool invocation order errors in complex workflows; Automating the refinement of agent execution strategies without manual engineering

EvoOntology: A Self-Evolving Ontology Layer for Data Agents

EvoOntology is a self-evolving ontology layer designed to bridge the gap between data agents and heterogeneous data sources.

ProblemThe 'agent-data gap' where agents can only access heterogeneous data through generic tools, leading to poor performance on large or complex data sources because existing static semantic layers do not scale or adapt to ag

Use it forEnabling LLM agents to query and reason over large, heterogeneous databases and file systems; Automating the construction and maintenance of semantic layers for data analytics tasks; Improving the accuracy of natural language to SQL or data retrieval tasks in complex environments

hey-llm-you-okay

A CLI tool for testing LLM applications that uses a cost-ordered pyramid of tests to verify behavior and quality.

ProblemLLM test suites often report green results without verifying actual behavior, and when tests fail, it is difficult to determine if the cause is a developer's code change or a silent update/drift in the underlying model p

Use it forCI/CD pipelines for LLM applications to gate deployments on deterministic and quality checks; Debugging failing LLM tests to determine if the cause is a code change or provider model drift; Verifying that the prompt used in tests matches the prompt sent in production

Beyond Token-Local Imitation: Reward-Compatible Temporal Credit Assignment for On-Policy Distillation

This paper proposes $\gamma$OPD, a method for on-policy distillation that uses discounted temporal credit assignment to balance long-horizon supervision with optimization stability.

ProblemExisting on-policy distillation objectives face a trade-off: token-level methods are stable but provide only local supervision, while sequence-level methods capture future credit but suffer from high variance dependent o

Use it forPost-training large language models for mathematical reasoning; Post-training large language models for code generation; Multi-teacher distillation scenarios where teacher and student model sizes differ

Prompt Guard

A Python library providing composable prompt-safety guards using TF-IDF, RAG, and LLM judges.

ProblemLack of lightweight, composable tools for detecting prompt injection and unsafe content in LLM applications.

Use it forFiltering user inputs for prompt injection attempts; Screening prompts for sensitive or prohibited content; Building custom safety layers for LLM applications

MASER: Multi-Agent Reinforcement Learning with Subgoals

This repository provides a Python implementation of the MASER algorithm, a multi-agent reinforcement learning method that generates subgoals from an experience replay buffer.

ProblemMulti-agent reinforcement learning often suffers from credit assignment and non-stationarity; MASER addresses this by generating subgoals from experience to guide agent learning.

Use it forResearching multi-agent reinforcement learning algorithms; Training agents in StarCraft II Multi-Agent Challenge scenarios; Implementing hierarchical reinforcement learning with subgoal generation

Curious Replay for DreamerV2

This repository provides a TensorFlow 2 implementation of Curious Replay, a method for prioritizing experience replay in model-based reinforcement learning agents.

ProblemStandard experience replay in model-based agents often fails to keep the world model up-to-date in changing environments, leading to poor action selection as the environment evolves.

Use it forTraining model-based agents in non-stationary environments where the world model needs to adapt quickly; Improving sample efficiency in reinforcement learning by prioritizing informative experiences for world model training; Researching the impact of curiosity-driven prioritization on agent performance in tasks like Crafter and DM Control

Full Duplex User Pairing via DQN

A reinforcement learning approach using Deep Q-Networks (DQN) to solve user pairing problems in full-duplex communication systems.

ProblemSolving the complex combinatorial optimization of user pairing in full-duplex systems efficiently and effectively.

Use it forOptimizing resource allocation in full-duplex wireless networks; Researching RL-based scheduling for uplink/downlink user pairing

Curious Replay for DreamerV3

A JAX implementation of the Curious Replay method integrated into the DreamerV3 model-based reinforcement learning agent.

ProblemStandard experience replay in model-based RL agents may not prioritize the most informative experiences, leading to slow or ineffective adaptation when the environment changes.

Use it forTraining model-based RL agents in non-stationary environments; Improving adaptation speed for DreamerV3 agents; Benchmarking curiosity-driven replay strategies in Crafter and DMC tasks

OpenEvals4J

A Java library for evaluating RAG systems and AI agents using LLM-as-a-Judge.

ProblemLack of a flexible, Java-native library for LLM-based evaluation of RAG systems and agents.

Use it forEvaluating the quality of Retrieval-Augmented Generation (RAG) pipelines; Assessing the performance of conversational AI agents; Measuring response faithfulness and completeness in LLM applications

mowa-eval

A CLI tool and GitHub Action for testing and evaluating LLM prompts.

ProblemPrompts are often treated as throwaway text without formal testing, leading to undetected regressions when they are modified.

Use it forBlocking pull requests when prompt changes cause performance regressions; Automating the generation of test cases for prompt files; Tracking prompt quality over time in CI/CD pipelines

Latency-Aware Orchestration for Multi-Agent LLM Workflows on Heterogeneous GPUs

This paper presents a prediction-guided runtime system for scheduling multi-agent LLM workflows on heterogeneous GPU pools.

ProblemExisting schedulers fail to account for future dependencies, model lifecycle actions, and time-varying resource availability in heterogeneous GPU pools, leading to suboptimal latency and resource utilization for multi-ag

Use it forScheduling complex multi-agent LLM workflows on shared GPU clusters; Optimizing model loading and placement in heterogeneous GPU environments; Reducing tail latency for bursty LLM serving workloads

FlowTT: Exploiting Computation Flow Reuse in Irregular Tensor-Train Embedding

FlowTT is a GPU execution framework designed to optimize Tensor-Train (TT) decomposition for embedding tables in recommendation models.

ProblemExisting TT-based embedding lookups are inefficient due to repeated materialization of intermediate results off-chip and failure to fully reuse partially shared computation flows across input indices, leading to high mem

Use it forAccelerating inference and training for large-scale recommendation systems using Tensor-Train compressed embeddings; Reducing GPU memory footprint and latency in models with irregular, skewed embedding lookup workloads

AutoRubric

AutoRubric is a Python library that standardizes the evaluation of LLM and VLM outputs using weighted rubrics and LLM-as-a-judge.

ProblemLacks a standard, reusable library for implementing weighted, LLM-based rubric evaluation, forcing developers to manually construct prompts and aggregation logic for each evaluation task.

Use it forScoring generated text against specific technical requirements with weighted criteria; Aggregating scores from multiple LLM judges to reduce variance in evaluation; Evaluating LLM outputs for factual accuracy using negative weights for hallucinations or incorrect claims

Every Kernel Is a Join: Automatic Multi-GPU Parallelism for AI Computations in Einsummable

Distributing an AI computation across the GPUs of a multi-GPU server is one of the central problems in systems-for-AI.

Para-Pipe: Exploiting Hierarchical Operator Parallelism of ML Computational Graphs on SoCs

As edge-based deep learning applications become more complex, optimizing performance on heterogeneous System-on-Chips (SoCs) presents unique challenges.

protectai/llm-guard

The Security Toolkit for LLM Interactions.

allenai/vla-evaluation-harness

One framework to evaluate any VLA model on any robot simulation benchmark.

modeltest: Personal LLM Engineering Maintenance Evaluation Harness

A frozen, self-hosted evaluation suite that measures LLM performance on a complex, multi-module Python and ESP32 firmware maintenance task.

ProblemStandard coding benchmarks often fail to capture the complexity of maintaining large, multi-module codebases with mixed technology stacks (Python + Embedded C).

Use it forBenchmarking specific LLM models on realistic software engineering maintenance tasks; Evaluating an LLM's ability to fix broken code in a multi-module project; Testing LLM performance on embedded firmware (ESP32) and backend API integration

huggingface/datasets

🤗 The largest hub of ready-to-use datasets for AI models with fast, easy-to-use and efficient data manipulation tools.

lm-evaluation-harness

A unified framework for evaluating large language models on a wide variety of academic benchmarks.

ProblemLack of a unified, reproducible, and efficient tool for running diverse benchmarks across different model types and inference engines.

Use it forBenchmarking LLMs on standard academic datasets like HellaSwag or ARC; Evaluating the performance of custom fine-tuned models; Comparing different model architectures or quantization methods

OmniVoice

OmniVoice is a massively multilingual zero-shot text-to-speech (TTS) model supporting over 600 languages.

ProblemLack of zero-shot TTS models with broad language coverage (600+) that also offer high inference speed and fine-grained voice control.

Use it forGenerating speech in over 600 languages with zero-shot voice cloning; Designing synthetic voices with specific attributes like gender, age, and pitch; High-speed TTS inference for real-time applications

Archon

Archon is a workflow engine for AI coding agents that allows developers to define deterministic development processes using YAML files.

ProblemAI coding agents produce inconsistent results, often skipping planning, forgetting to run tests, or ignoring templates, leading to unreliable and non-repeatable workflows.

Use it forAutomating the full lifecycle from bug fix to pull request creation; Running parallel code fixes in isolated git worktrees; Enforcing strict validation gates and code review steps in AI-assisted development

memex

memex is a local-first, zero-dependency library for Python and JavaScript that provides persistent, searchable memory for LLMs using a single SQLite file.

ProblemLLMs lack persistent memory, forcing users to re-explain context, preferences, and project details in every new session.

Use it forAdding persistent user preference memory to a chatbot; Creating a local, private memory layer for personal AI assistants; Implementing cross-session context retention for LLM applications

Contrastive Explanations in Quantitative Bipolar Argumentation Frameworks

Argumentation frameworks are useful tools for representing and reasoning with information in a variety of settings, e.g.

mini-datasets

A repository containing summarized versions of autonomous driving datasets, specifically extracting small, fast-to-process subsets from large sources like the Waymo Open Dataset.

ProblemStandard autonomous driving datasets (like Waymo) are extremely large (e.g., 400MB per record), making fast iteration and local development difficult due to high download and processing times.

Use it forRapid prototyping of motion planning algorithms; Visualizing traffic scenarios with lightweight data; Iterating on autonomous driving models without heavy I/O overhead

Attention Is All You Need

This paper introduces the Transformer, a neural network architecture that relies entirely on self-attention mechanisms to process sequences, eliminating the need for recurrent or convolutional layers.

ProblemPrevious sequence transduction models relied on recurrent or convolutional networks that were difficult to parallelize and required significant training time.

Use it forMachine translation; Constituency parsing; General sequence transduction tasks

adibhatt1997/agent-cordon

Zero-dependency prompt-injection and data-exfiltration firewall for LLM agents: scans untrusted text and outbound actions, sees through obfuscation, guards the MCP/tool boundary.

Mercury Agent

Mercury is a soul-driven AI agent that runs 24/7 via CLI or Telegram, featuring permission-hardened tools, token budgets, and persistent SQLite-backed memory.

ProblemStandard AI agents often lack persistent memory, execute dangerous commands without explicit permission checks, and are difficult to run as always-on background services with multi-channel access.

Use it forRunning a persistent personal AI assistant that remembers user preferences and goals; Executing shell commands and file operations with strict permission controls and approval flows; Interacting with an AI agent via Telegram for mobile access

AegisTrace

AegisTrace is a Go-based verifiable runtime for durable AI agents that compiles YAML workflows into sealed artifacts and records every execution effect in a hash-chained journal.

ProblemAI agents are non-deterministic and lack verifiable execution records, making it difficult to audit, debug, or prove the integrity of their actions and state transitions.

Use it forAuditing AI agent actions for compliance and security; Debugging non-deterministic AI workflows via offline replay; Proving the integrity of specific events in an AI execution log

Vecgo

Vecgo is a pure Go, embeddable hybrid vector database that combines HNSW and DiskANN indexing with commit-oriented durability.

ProblemLack of a pure Go, embeddable vector database with production-grade features like durability, MVCC, and hybrid search without external dependencies.

Use it forHigh-performance vector search in Go applications; Embedding a vector database directly into Go services; Hybrid search combining BM25 and vector similarity

philippgille/chromem-go

Embeddable vector database for Go with Chroma-like interface and zero third-party dependencies.

Agentic-Flow-AI

An open-source framework for building autonomous multi-agent AI workflows.

ProblemLinear prompt-based AI systems struggle with complex, long-horizon tasks that require coordination, memory, and dynamic tool usage.

Use it forBuilding complex, long-horizon AI tasks that require coordination between multiple specialized agents; Implementing human-in-the-loop guardrails for autonomous AI systems; Managing token costs in multi-agent workflows

biomni_bench

A research project and evaluation suite for measuring the reliability of autonomous biomedical AI agents like Biomni, GenoMAS, and AutoBA.

ProblemAutonomous biomedical agents can produce inconsistent or silently wrong results across repeated runs, and there is no standardized way to measure or improve this reliability.

Use it forBenchmarking the reliability of new biomedical LLM agents; Identifying stable-wrong vs. unstable-recoverable agent behaviors; Training improved agents via trajectory-ensemble distillation

Agent-Aegis

Agent-Aegis is a Python library that provides a unified governance layer for AI agents, automatically instrumenting frameworks like LangChain, CrewAI, and OpenAI to enforce policies.

ProblemThe lack of a unified, framework-agnostic method for implementing security guardrails, policy enforcement, and auditing across the fragmented ecosystem of AI agent frameworks.

Use it forEnforcing security policies across multiple agent frameworks simultaneously; Detecting and blocking prompt injection attacks in real-time; Masking PII in agent inputs and outputs

Promptfoo GitHub Action

A GitHub Action that integrates Promptfoo into CI/CD pipelines to evaluate LLM prompts, agents, and RAG systems.

ProblemLacks a standardized, automated way to test and monitor LLM prompt performance and safety within existing CI/CD workflows.

Use it forAutomated regression testing of LLM prompts in pull requests; Continuous monitoring of agent performance and safety; Comparing performance across different LLM providers (OpenAI, Anthropic, etc.) in CI

TDDF: Behaviour Regression Tests for AI Agents

TDDF is a local-first testing framework that runs deterministic behaviour regression tests on AI agents using YAML-defined attack scenarios.

ProblemExisting AI agent security testing often relies on non-deterministic LLM judges or cloud services, making it difficult to create reproducible, local, and compliance-ready regression tests for agent behavior.

Use it forDetecting prompt injection vulnerabilities in LangGraph or OpenAI Agents SDK applications; Verifying that agents do not exfiltrate secrets via hidden web content or document masking; Generating compliance evidence (JUnit XML) for EU AI Act or ISO 42001 audits

EvalShift CLI

EvalShift is a Python CLI tool for LLM migration and regression testing that replays captured agent behavior against candidate models.

ProblemMigrating between LLM versions typically requires guessing which behaviors changed, lacking a standardized way to detect regressions or validate that a new model performs equivalently to the old one.

Use it forComparing performance between two LLM versions before migrating an agent; Detecting tool-call regressions during model updates; Gating model changes in CI/CD pipelines using statistical significance

llm-runtime

A production-oriented Kubernetes platform for shared local LLM inference and a trusted OpenAI-compatible gateway for subscription providers.

ProblemConsumer projects typically duplicate effort in managing GPU placement, model serving, and provider credentials; this project externalizes those infrastructure concerns into a stable service contract.

Use it forDeploying shared local LLM inference clusters; Accessing subscription-based AI providers via a unified gateway; Managing runtime telemetry and network policies for AI services

BUPT-GAMMA/CPF

The official code of WWW2021 paper: Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework.

HeterSumGraph: Heterogeneous GNN for Document Summarization

This repository provides the official PyTorch implementation of the ACL 2020 paper 'Heterogeneous Graph Neural Networks for Extractive Document Summarization'.

ProblemStandard extractive summarization methods often fail to capture complex, type-specific relationships between sentences (e.g., topic vs. entity mentions) that are better represented in heterogeneous graphs.

Use it forAcademic research on graph neural networks for natural language processing; Extractive document summarization for news articles; Reproducing baseline results for summarization benchmarks

PositionalHidden

An implementation of a method to mitigate position bias in Large Language Models by scaling a single dimension of the hidden states.

ProblemLLMs suffer from position bias, particularly in long-context scenarios, leading to degraded performance when relevant information is located in the middle of the input sequence.

Use it forImproving LLM performance on long-context tasks like multi-document QA; Mitigating the 'lost in the middle' phenomenon in LLMs; Evaluating the impact of positional encoding modifications on model accuracy

Esha-Mirza/skitrack

A lightweight, developer-friendly ML experiment tracking library that automatically records model runs, parameters, metrics, training time, and dataset information using local storage.

RAGCheck

RAGCheck is a Python library and CLI tool for evaluating Retrieval-Augmented Generation (RAG) pipelines.

ProblemExisting RAG evaluation tools often rely on unvalidated LLM judges and lack mechanisms to verify judge reliability, measure refusal behavior, or integrate easily into CI/CD pipelines for regression detection.

Use it forValidating LLM judges against human labels before trusting automated metrics; Detecting hallucinations and refusal calibration issues in RAG systems; Running CI regression checks to ensure code changes do not degrade RAG quality

A Blind Trust, the Bloody Thrust: When Attacker-Controlled Hook Updates Steer AI Agent Harnesses towards Malicious Behaviors

Modern AI agent harnesses expose lifecycle hooks that bind shell commands to runtime events such as session start, tool calls, and file edits.

llm-judge-audit

A pre-registered research protocol and codebase for auditing the reliability of LLM-as-judge systems.

ProblemUncertainty about the reliability and bias of LLM-as-judge systems, and the lack of a cost-effective strategy for combining automated and human evaluation.

Use it forEvaluating the reliability of LLM judges for instruction-following tasks; Determining the optimal balance between cheap automated judges and expensive human review; Measuring position, length, and self-preference biases in LLM evaluations

sigil: Error bars for LLM-as-judge evals

A dependency-free TypeScript library and CLI tool that computes statistical reliability metrics (calibration, finite-sample risk certificates, drift monitoring) for LLM-as-judge evaluations.

ProblemLLM-as-judge evaluations often lack statistical rigor, relying on single accuracy figures without error bars or calibration checks, which makes it difficult to trust cost-savings claims or detect model drift.

Use it forAuditing the reliability of LLM-as-judge evaluation pipelines; Determining optimal model routing policies based on cost-quality trade-offs with statistical guarantees; Monitoring drift in automated evaluation systems over time

AI-Assisted Design of a Post-Quantum Cryptographic Accelerator

This paper presents a case study on using an agentic LLM to design a post-quantum cryptographic accelerator (ML-KEM-768 and ML-DSA-65) on a Kintex-7 FPGA.

ProblemStandard known-answer tests (KATs) fail to detect defects in message-dependent execution paths, such as the ML-DSA rejection loop, allowing flawed silicon to ship.

Use it forDesigning post-quantum cryptographic hardware accelerators; Verifying cryptographic implementations against FIPS standards; Using agentic LLMs for RTL design and hardware bring-up

PatchBench: Evaluating AI Agents for Vulnerability Patching

PatchBench is a new benchmark designed to evaluate AI agents on realistic C/C++ vulnerability patching tasks.

ProblemExisting vulnerability patching benchmarks often validate patches only by checking if a specific Proof-of-Concept (PoC) crash is suppressed, allowing agents to pass via memorized historical patches or surface-level fixes

Use it forEvaluating the robustness of AI coding agents on security-critical tasks; Benchmarking AIxCC participants against more rigorous validation criteria; Researching the gap between crash suppression and root-cause fixing in automated repair

Sentinel-RL: Offloading Topological Reasoning from LLM Agents in the Security Operations Center

Sentinel-RL is an agentic-SOC architecture that decouples topological reasoning from semantic reasoning to improve reliability in enterprise security operations.

ProblemLLM agents lack the context window to hold large authentication graphs and cannot guarantee that recommended containment actions are consistent with the underlying network topology.

Use it forAutomated detection and investigation of lateral movement in enterprise networks; Generating analyst-readable narratives for security alerts; Managing containment actions for red-team events in HPC clusters

InstantID-Rome

InstantID-Rome is an improved version of the InstantID model designed for identity-preserving image generation.

ProblemOriginal InstantID requires manual face keypoint input, produces unnatural facial saturation, has rigid body composition, and offers limited editability and layout stability.

Use it forGenerating realistic portraits of a specific person in various scenarios without manual face landmarking; Editing identity-preserving images with text prompts (e.g., adding accessories or changing age); Creating consistent character designs across different poses and backgrounds

DataTonic

DataTonic is a multi-agent orchestration framework that combines Autogen, Semantic Kernel, and TaskWeaver to create nested agent swarms for data processing.

ProblemThe slow, manual, and expensive process of creating business intelligence assets, databases, and reports, which often requires specialized data engineering skills and long review cycles.

Use it forGenerating sales profiles and strategies from raw data; Creating business intelligence databases and reports; Automating junior-level data processing and document creation tasks

nano-step/eval-harness

A behavior-regression testing harness for LLM agents, specifically designed for opencode skills.

ProblemLLM agents and skills are non-deterministic, making it difficult to distinguish between real regressions, flaky behavior, and environmental issues during development and CI.

Use it forDetecting behavior regressions in opencode skills during git push; Attributing test failures to specific causes like skill changes or environment issues; Managing LLM evaluation costs via tiered smoke and full testing modes

EvalCore

EvalCore is a single-binary evaluation runner for LLM applications and agents that uses snapshot testing to detect regressions.

ProblemLLM applications are non-deterministic and expensive to test, making it difficult to reliably detect regressions in CI/CD pipelines or locally without incurring significant API costs and flaky test results.

Use it forGating pull requests in CI/CD pipelines to block LLM behavior regressions; Testing local shell commands or HTTP APIs for LLM-backed applications; Replaying recorded model calls to ensure deterministic testing without API costs

Chronicle: Record-and-Replay for Agent Decision Graphs

Chronicle is a Python library that records immutable 'Envelope' traces of agent decision points (LLM calls, tool calls, and routing) to enable deterministic regression testing.

ProblemTesting agent behavior is currently expensive, slow, and non-deterministic due to reliance on live LLM calls, making it difficult to reliably catch regressions in control flow and tool safety.

Use it forReproducing a production agent failure as a committed regression test; Verifying fixes for control-flow and tool-safety regressions without live API calls; Performing deterministic replay of multi-agent orchestration logic in CI pipelines

Agentic Harness Engineering (AHE)

AHE is a Python framework for automatically evolving coding-agent harnesses by optimizing prompts, tools, and middleware based on observability data.

ProblemCoding agents often underperform on complex tasks due to suboptimal harness configurations (prompts, tools, context handling); manual tuning is slow, non-scalable, and doesn't generalize well across models.

Use it forImproving the performance of coding agents on benchmarks like Terminal-Bench 2; Automating the refinement of agent system prompts and tool definitions; Transferring optimized agent configurations across different base models

Augment SWE-bench Verified Agent

An open-source implementation of a coding agent designed to solve SWE-bench Verified tasks, utilizing Claude Sonnet 3.7 as the core driver and OpenAI o1 for ensembling.

ProblemLack of a simple, open-source, and high-performing implementation for SWE-bench that leverages current state-of-the-art models for complex codebase navigation and regression testing.

Use it forEvaluating LLM performance on real-world software engineering tasks; Automating the resolution of GitHub issues in open-source projects; Experimenting with agentic coding workflows using off-the-shelf models

AutoCodeRover

AutoCodeRover is an autonomous software engineering agent that resolves GitHub issues by combining LLMs with program structure-aware code search and statistical fault localization.

ProblemManual software debugging and patch generation are time-consuming and expensive, especially in large, complex codebases where locating the correct fix location is difficult.

Use it forAutomatically fixing bugs in open-source repositories; Implementing feature requests from GitHub issues; Performing statistical fault localization in large codebases

OpenGraphMemory

A self-hosted platform for extracting knowledge graphs from documents and codebases, storing them in a temporal PostgreSQL database, and providing persistent memory for AI agents.

ProblemAI agents lack persistent, structured memory for long-term context, and existing tools often fail to provide evidence-backed, temporal tracking of knowledge extracted from documents and code.

Use it forBuilding persistent long-term memory for AI agents; Extracting structured knowledge graphs from codebases for code understanding; Creating searchable, evidence-backed knowledge bases from unstructured documents

judge-reliability

A research repository that calibrates LLM-as-judge reliability using 1,814 human-labeled MT-Bench comparisons.

ProblemThe lack of standardized calibration for LLM judges, leading to over-reliance on potentially biased or unstable automated evaluations without understanding their limits relative to human judgment.

Use it forCalibrating LLM judges before deploying them for automated evaluation; Benchmarking the reliability of different LLM models as judges; Analyzing the impact of prompt sensitivity and presentation order on LLM judgments

LLM-as-a-Judge Evaluation Framework

A Python framework for evaluating LLM outputs using automated judges with structured rubrics.

ProblemManual evaluation of LLM outputs is slow and expensive, while naive automated LLM judges suffer from position bias and can be fooled by confidently incorrect responses.

Use it forEvaluating the quality of LLM responses against specific criteria like correctness and completeness; Comparing two different LLM models or prompts to determine relative performance; Validating the reliability of an automated LLM judge against human expert labels

LLM Bias Evaluation Framework

A model-agnostic pipeline for measuring and mitigating four specific biases (verbosity, score inflation, position, and method disagreement) in LLM-as-a-Judge systems.

ProblemLLM-as-a-Judge systems are often trusted blindly, but they suffer from systematic biases (like preferring longer answers or the first option) that can invalidate evaluation results.

Use it forAuditing LLM-based evaluation systems for systematic biases; Comparing the reliability of rubric-based vs. pairwise comparison methods; Calibrating LLM judges to reduce score variance and inflation

OpenClaw Memory

A three-layer persistent memory system for AI agents using bash scripts, organized into a knowledge graph, daily notes, and tacit knowledge files.

ProblemSolves the problem of AI agents lacking reliable, structured, and persistent memory that can handle fact changes and temporal context without relying on opaque embedding similarity.

Use it forManaging long-term context for autonomous agents; Tracking relationships and historical facts about specific entities; Automated maintenance of agent memory without LLM embeddings

Agentic Knowledge Graphs over PubMed

A research repository demonstrating a retrieval-augmented generation system for biomedical questions that uses a massive citation and ontology knowledge graph built from structured metadata rather than LLM extraction.

ProblemStandard RAG systems often fabricate answers without signaling uncertainty, making them unsafe for clinical or regulatory use; this project addresses the need for legible entitlement to answer and traceable provenance.

Use it forAnswering biomedical research questions with traceable provenance; Demonstrating cost-effective knowledge graph construction for large scientific corpora; Evaluating retrieval-augmented generation systems with calibrated abstention

Meta Knowledge Graph (MKG)

A self-improving, graph-structured memory layer for AI agents backed by Neo4j.

ProblemAI agents typically lack persistent memory, forcing users to repeatedly provide context or losing track of past decisions and user preferences between sessions.

Use it forPersisting project-specific context and decisions across AI agent sessions; Maintaining a personalized user profile that adapts agent behavior over time; Providing episodic memory of past work windows to AI coding assistants

Pile Dedupe

A Python script collection for deduplicating the Pile dataset using MinHash and Locality-Sensitive Hashing (LSH).

ProblemLarge text datasets like the Pile contain many near-duplicate documents, which can lead to data leakage and overfitting if not properly deduplicated.

Use it forDeduplicating the Pile dataset before training language models; Identifying near-duplicate documents in large text corpora; Generating a clean, deduplicated data stream for model training

KnowFeat: Knowledge-Guided Feature Engineering via LLM Agents

Automated feature engineering with large language models (LLMs) can produce semantically meaningful features for tabular data, yet existing methods lack structured domain knowledge, rigorous verification, and explainable.

Semantic Bayesian World Models

This paper proposes Semantic Bayesian World Models (SBWMs) to bridge the gap between crisp knowledge graphs and probabilistic foundation models.

ProblemThe mismatch between crisp knowledge graph assertions and the native probabilistic reasoning of foundation models, which currently limits their integration to simple data-feeding pipelines.

Use it forHome-security agents distinguishing between couriers and burglars using probabilistic reasoning; Aggregating actuarial estimates via entailment rather than string frequency; Solving planning tasks that language models typically fail

NIKHIL-KASHMEERABEN-RUPALA/Neural_Forge

NeuralForge is an end-to-end LLM fine-tuning & serving platform.

Neal006/memorylens-bench

The open-source benchmark for LLM memory decay.

Tiny QA Benchmark++

Tiny QA Benchmark++ is a micro-benchmark suite and Python package for rapid LLM evaluation, featuring a 52-item immutable gold dataset and a CLI tool for generating synthetic multilingual QA packs.

ProblemStandard LLM evaluation benchmarks are often too large and slow for frequent, iterative testing in CI/CD environments, making it difficult to catch regressions quickly.

Use it forRapid smoke testing of LLMs in CI/CD pipelines; Regression testing for prompt engineering changes; Generating custom multilingual QA datasets for specific domains

ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning

ARISE-RL is a reinforcement-learning training framework for open-ended agents.

ProblemOpen-ended agentic RL lacks verifiable gold answers and scalable rubrics, and long-horizon tasks often produce brittle, unstable rewards that create weak or noisy optimization signals for group-based policy learning.

Use it forTraining open-ended LLM agents that use tools over long horizons; Generating intermediate-difficulty tasks and rubrics aligned to a solver's evolving capability; Providing fine-grained rubric satisfaction signals for multi-step reasoning and tool use

Synthetic Semantic Supervision for Small Code Transformers

An arXiv empirical study of contrastive pretraining for compact code transformer encoders.

ProblemExisting compact code encoders depend on labor-intensive, inconsistent human docstrings or costly, setting-specific execution traces.

Use it forcode search and retrieval; code classification; code generation assistance

Srivatsa03/rag-redteam

Red-team your RAG pipeline for prompt injection, source-document leakage, and cross-document smuggling, in CI.

Computable Representation of the Physical Laboratory

This paper proposes a formal framework for representing physical laboratories as computable objects with typed states and capability-bound operations.

ProblemThe lack of a formal, computable representation of the physical laboratory that allows AI agents to safely plan, verify, and execute complex experimental workflows with explicit state dependencies.

Use it forAutomating complex multi-step laboratory experiments; Verifying preconditions and constraints before executing physical operations; Enabling end-to-end autonomous scientific discovery workflows

Magnitude

Magnitude is an open-source inference server that profiles local hardware to recommend and run the best local LLMs for agent workloads.

ProblemDevelopers using AI agents face high API costs, privacy risks from sending code to the cloud, and the complexity of manually configuring local LLMs (Ollama, etc.) to match their specific hardware constraints.

Use it forRunning coding agents like Claude Code or Codex locally without API costs; Executing private, offline LLM inference for sensitive codebases; Automatically selecting and tuning LLMs based on available RAM and CPU/GPU capabilities

Environment Evolution for Terminal Agents

Scaling interactive and verifiable environments is critical for training terminal agents.

radixark/miles

Miles is an enterprise-facing reinforcement learning framework for LLM and VLM post-training, forked from and co-evolving with slime.

CROCODIL: Cross-Model Code Editing with LLMs

Large language models (LLMs) have become ubiquitous tools for code generation and editing.

LabelMate: An LLM-Driven Framework for Refined Issue Report Labeling

Software users often submit issue reports to a product's issue tracking system to report defects, suggest enhancements, or raise other product-related concerns.

When Models Edit Too Much: On the Fidelity of Minimal Code Edits

Large language models (LLMs) are increasingly used to edit existing code, but correctness alone is not enough: useful repairs should also be minimal, reviewable, and faithful to the original implementation.

DRACO: Fine-Grained Credit Assignment with Dynamic Rubrics for Long-Horizon Agent Training

Reinforcement Learning from Verifiable Rewards works well when a task has a programmatic checker, but most long-horizon agent domains have none.

ATIBA: Grounded Integrity and Quality Checking for Research Papers

Checking a manuscript's reference integrity, its compliance with a target venue's specific submission rules, and its adherence to community reporting standards is manual, repetitive, and different for every venue so in p.

SWE-Gate: Passing Functional Tests Is Not Enough for Software Engineering Agents

Repository-level software engineering benchmarks have significantly advanced the evaluation of coding agents, but existing benchmarks primarily measure whether generated patches pass functional tests and overlook review-.

VecturaKit

VecturaKit is a Swift-based vector database library designed for on-device Retrieval-Augmented Generation (RAG) applications.

ProblemLack of a native, Swift-first vector database solution for on-device RAG that integrates seamlessly with Apple's frameworks (NaturalLanguage, MLX) and supports hybrid search without relying on external cloud services or

Use it forBuilding offline-capable RAG features for iOS and macOS apps; Implementing local semantic search in Apple ecosystem applications; Creating on-device document retrieval systems without cloud dependencies

mlx-embeddings

A Python library for running Vision and Language Embedding models locally on Apple Silicon Macs using the MLX framework.

ProblemLack of a unified, efficient library for running diverse embedding models locally on Apple Silicon using MLX.

Use it forLocal semantic search on Mac hardware; Multimodal retrieval using text and image embeddings; Running embedding models without cloud API dependencies

mlx-tune

mlx-tune is a Python library that enables fine-tuning of Large Language Models (LLMs), Vision, Audio, and OCR models natively on Apple Silicon using the MLX framework.

ProblemMac users cannot use Unsloth (the gold standard for efficient LLM fine-tuning) because it relies on Triton, which is not supported on Apple Silicon; this creates a workflow friction where local prototyping requires a dif

Use it forPrototyping LLM fine-tuning locally on a MacBook before scaling to cloud GPUs; Fine-tuning vision models like I-JEPA and V-JEPA on Apple Silicon; Training TTS and STT models (e.g., Whisper, Qwen3-TTS) locally

Adaptive KV Retention for LLM Agents at Human-Approval Timescales

This paper addresses the performance degradation in LLM serving systems when agent requests are suspended for long periods (minutes to hours) awaiting human approval.

ProblemExisting LLM serving systems are optimized for seconds-scale pauses; long suspension times for human approval cause either massive GPU memory waste (if KV is retained) or high latency (if KV is evicted and recomputed), l

Use it forServing LLM agents that require human-in-the-loop approval steps; Optimizing GPU utilization in multi-tenant LLM inference clusters with variable request latencies; Reducing resume latency for suspended agentic workflows without sacrificing throughput for active requests

The Irreversibility Budget: Fleet-Level Risk Accounting for Agent OS

This paper proposes the 'irreversibility budget,' a mechanism for managing cumulative risk in fleets of LLM agents by treating irreversibility as a first-class resource.

ProblemCurrent controls check effects one at a time, allowing a fleet of individually authorized agents to overdraw a principal's risk under a shared trigger while every local gate remains correct.

Use it forPreventing fleet-level risk overdraw in multi-agent systems; Implementing admission control for autonomous agents performing irreversible actions; Managing shared risk limits across multiple agents and tenants

Analog-DB: An Agent-First Analog Integrated Circuit Database

Analog-DB is an open-source, versioned database for analog integrated circuit designs that uses a domain-specific language to capture process-neutral topologies, reusable testbenches, and machine-readable datasheets.

ProblemAnalog IC designs are difficult to share because foundry non-disclosure agreements restrict process details and published testbenches are rarely released, hindering reuse and verification.

Use it forAI agents automatically discovering and reusing verified analog circuit blocks; Retargeting analog designs across different foundry process kits; Automated sizing and verification of analog circuits using machine-readable specifications

Examining the Vulnerability of Multi-Agent Medical Systems to Human Interventions for Clinical Reasoning

This paper analyzes how human interventions at specific 'fault points' in multi-agent medical systems affect diagnostic accuracy.

ProblemMulti-agent medical systems are vulnerable to external influence and cognitive biases, leading to diagnostic errors and drift.

Use it forImproving diagnostic robustness in AI-assisted medical diagnosis; Identifying vulnerable reasoning steps in multi-agent clinical workflows; Mitigating cognitive biases in automated medical reasoning systems

vLLM Semantic Router

A programmable routing layer for building Mixture-of-Models systems across heterogeneous LLM infrastructure.

ProblemHeterogeneous LLM infrastructure requires complex, hard-coded routing logic to balance cost, quality, latency, and privacy constraints across different models and compute locations.

Use it forRouting LLM requests to the most cost-effective model based on complexity; Directing sensitive data to private or edge models for privacy compliance; Composing multiple models to balance quality and latency for specific workloads

Post-Training Ternarization of Qwen3-4B Capability, Effective Bit Budget, Storage Compression, and Deployment

Ultra-low-bit language models can reduce storage and memory bandwidth, but a nominal "1.58-bit" label does not fully describe the stored representation, retained capability, or runtime behavior.

Door-in-the-Face Requests and Refusal Behaviour in Large Language Models

Does the door-in-the-face technique work on language models? In humans, a large request that is refused makes a smaller follow-up request more likely to be granted.

Beyond Outcome Gaps: Process-Aware Fairness Diagnosis for LLM-based Multi-Agent Decision Systems

LLM-based multi-agent systems (MAS) are increasingly considered for high-stakes decision-making, yet outcome-based fairness audits can miss where risks arise within the decision trajectory.

FlowCraft

FlowCraft is a modular Go SDK for building AI applications with long-term memory, knowledge retrieval, and multi-provider support.

ProblemLack of a unified, provider-agnostic Go SDK for AI agents that supports long-term memory and modular architecture.

Use it forBuilding Go-based AI agents with persistent memory; Creating multi-provider LLM applications without vendor lock-in; Developing interactive AI workflows with a TUI

jieyefriic/rp-engine

YAML-native agent workflow execution engine, written in Rust.

AgenticArXiv-RL

A reinforcement learning pipeline for training ReAct agents to perform arXiv paper retrieval tasks.

ProblemTraining LLM agents for complex tool-use tasks typically requires expensive human annotation for preference data or complex reward models; this project solves that by using rule-based, verifiable rewards and a determinis

Use it forTraining LLMs to autonomously search and download academic papers; Researching agentic reinforcement learning with verifiable rewards; Developing tool-use agents for scientific literature workflows

datacurve-ai/deep-swe

Measuring frontier coding agents on original, long-horizon engineering tasks.

TReS: No-Reference Image Quality Assessment via Transformers

A research repository implementing a No-Reference Image Quality Assessment (NRIQA) method.

ProblemExisting image quality metrics often require reference images or fail to align with human perceptual judgments in no-reference scenarios.

Use it forAutomated quality control for generated images; Benchmarking image generation models; Filtering low-quality assets in computer vision pipelines

CNNIQA

A PyTorch implementation of the CNNIQA model for no-reference image quality assessment, reproducing the CVPR 2014 paper.

ProblemManual human evaluation of image quality is subjective, expensive, and not scalable; existing metric-based methods often fail to capture complex perceptual distortions.

Use it forPredicting perceptual quality scores for images without reference ground truth; Evaluating image compression or processing algorithms using learned quality metrics; Researching deep learning approaches for image quality assessment

No-Reference Image Quality Assessment using BRISQUE Model

This repository provides an implementation of the BRISQUE (Basic Reference-free Image Spatial quality eXponent) model for no-reference image quality assessment.

ProblemLack of a simple, open-source implementation for calculating no-reference image quality scores using the BRISQUE model in Python and C++.

Use it forEvaluating the quality of compressed or distorted images without a reference; Automating image quality checks in computer vision pipelines; Comparing the performance of different image processing algorithms

MANIQA: Multi-dimension Attention Network for No-Reference Image Quality Assessment

MANIQA is a PyTorch implementation of a deep learning model for no-reference image quality assessment (NR-IQA).

ProblemExisting no-reference image quality assessment methods often fail to accurately predict quality scores for images with GAN-based distortions, which are common in modern generative AI workflows.

Use it forAutomated quality scoring of generated images in GAN pipelines; Benchmarking perceptual quality of image restoration models; Filtering low-quality images from datasets

LLM Intent & Entity Evaluation Framework

A Python framework that uses Large Language Models (via Vertex AI) to evaluate ASR models by measuring semantic intent and entity preservation rather than just word error rate.

ProblemTraditional ASR metrics like WER and CER penalize all textual variations equally, failing to distinguish between benign errors that preserve meaning and severe errors that break user intent or alter key entities.

Use it forEvaluating ASR model performance for Indic languages where minor spelling variations do not change meaning; Assessing the functional utility of voice command systems by checking if user intent is preserved; Comparing ASR models based on semantic accuracy rather than strict textual match

llm-eval-simple

A Python-based framework for benchmarking LLMs by sending prompts to OpenAI-compatible APIs and evaluating responses against expected answers.

ProblemLack of a lightweight, configurable tool for running batch evaluations and visualizing results for multiple LLMs against custom prompt/answer pairs.

Use it forBenchmarking accuracy and latency of multiple LLMs on a custom dataset; Comparing performance between different model versions or providers; Visualizing evaluation results in an interactive web dashboard

recall

An asynchronous Python library providing a memory pipeline for LLM agents.

ProblemLLM agents lack persistent, structured memory mechanisms to recall past interactions, learn procedures, or retrieve relevant semantic context efficiently.

Use it forBuilding long-term memory for autonomous LLM agents; Implementing hybrid search over agent interaction history; Managing procedural knowledge for AI assistants

Hermes Consolidating Local Memory

A local-first memory provider plugin for the Hermes Agent that uses SQLite and FTS5 for durable, private, and evidence-backed long-term memory.

ProblemLack of a local-first, durable, and privacy-preserving memory system for agents that can handle contradictions, temporal context, and crash recovery without cloud dependencies.

Use it forProviding isolated long-term memory for Hermes Agent instances; Storing and retrieving evidence-backed facts with provenance tracking; Managing episodic, procedural, and autobiographical memory locally without cloud services

MemFeelAI

MemFeelAI is a Python-based AI system that implements an 8-level hierarchical memory architecture (from sensor buffer to full-text search) and integrates real-time emotion analysis services.

ProblemStandard AI chatbots lack persistent, structured long-term memory and the ability to track or respond to user emotional states over time.

Use it forBuilding AI agents with long-term persistent memory; Analyzing emotional tone in voice and text inputs; Implementing hierarchical data retention with TTL-based expiration

ProofGatedAgents

A governed multi-agent framework that treats LLM outputs as untrusted suggestions, enforcing correctness through deterministic verification, locked governance contracts, and proof-gated execution stages.

ProblemCurrent agent frameworks trust non-deterministic LLM output and rely on retries, leading to silent drift, hidden writes, and runaway autonomy that are unsafe for production systems.

Use it forBuilding auditable autonomous systems where LLM actions must be provably correct; Creating reproducible multi-agent pipelines with strict policy enforcement; Implementing deterministic verification gates for non-deterministic AI components

CRB-Guided Sensing and Resource Allocation for Human Pose Prediction in ISCC Systems

This paper proposes a framework for integrated sensing, communication, and computation (ISCC) systems that jointly optimizes radar beamforming, model inference depth, and computation frequency to minimize human pose pred.

ProblemMinimizing pose prediction error in resource-constrained ISCC systems where sensing, communication, and computation resources must be jointly optimized.

Use it forIndoor human-centric applications requiring continuous pose tracking; Resource-constrained ISCC systems needing proactive resource allocation; Joint optimization of sensing and computation for improved prediction accuracy

Recon-GraphRAG

A domain-agnostic Python SDK for building knowledge graphs in Neo4j or Memgraph using the Microsoft GraphRAG philosophy.

ProblemExisting GraphRAG implementations are often tied to specific graph databases or lack the flexibility to support multiple search paradigms and pluggable LLM providers.

Use it forBuilding enterprise knowledge graphs from unstructured documents; Implementing retrieval-augmented generation with traceable citations; Performing hierarchical community analysis on graph data

Network-Aware Forecasting on Wireless Access Points

This paper defines 'network-aware deployability' for running predictive ML models on enterprise wireless access points (APs).

ProblemModels that perform well on standard development hardware often fail or degrade network services when deployed on resource-constrained access points due to CPU and memory contention with packet processing.

Use it forDeploying predictive maintenance models on enterprise Wi-Fi infrastructure; Evaluating the latency and throughput impact of edge AI on network services; Selecting appropriate model sizes for constrained AP hardware

SoK: Where Do Flow Labels Come From? Auditing Label Provenance in Encrypted Traffic Benchmarks

Encrypted traffic classification infers semantics beyond the flow record from transport-layer observables, and supervised training rests on labels that hold for the individual flow they are attached to.

WiP: Characterizing and Defending Against Mobile-Agent-Driven MFA Automation

Mobile agents automate smartphone tasks by interpreting interfaces, interacting with apps, and coordinating cross-app workflows.

AI Engineering Standards (AIES)

AIES is a Python-based evaluation framework that assesses AI models and repositories using versioned engineering scenarios to generate an Engineering Capability Matrix (ECM).

ProblemStandard AI leaderboards provide aggregate scores that do not indicate whether a model is safe or capable for specific, high-stakes engineering tasks like refactoring payment paths or reviewing security boundaries.

Use it forEvaluating whether an AI model is suitable for specific engineering tasks like API design or security review; Assessing the quality and governance of a software repository's architecture and testing practices; Comparing multiple AI model deployments to identify supported differences in capability

Eval-Forge

Eval-Forge is a self-hosted, open-source platform for evaluating Large Language Models using methods like G-Eval and LLM-as-a-Judge.

ProblemLacks a structured, self-hosted, and reproducible framework for continuous LLM evaluation, currently forcing teams to rely on ad-hoc scripts or expensive vendor solutions.

Use it forAutomated regression testing of LLM prompts and models in CI/CD pipelines; Benchmarking RAG systems for faithfulness and relevance; Comparing different LLM judges (G-Eval, DeepEval) on the same dataset

code2vec

A Python implementation of the code2vec algorithm for learning distributed representations of code.

ProblemTraditional code analysis relies on syntactic matching, which fails to capture semantic similarity between differently written but functionally equivalent code.

Use it forCode completion; Code retrieval; Semantic similarity search between code snippets

AutoComments

A research project implementing a deep neural network to automatically generate comments for Java code snippets.

ProblemDevelopers spend a significant portion of their time (approx 59%) on program comprehension; manual commenting is time-consuming and often incomplete.

Use it forGenerating descriptive comments for Java functions during software maintenance; Assisting developers in program comprehension tasks; Researching the application of AST-based encodings for code summarization

code2vec

A TensorFlow implementation of the code2vec neural network for learning distributed representations of code.

ProblemLack of effective neural network models for learning semantic representations of source code that can generalize to tasks like name prediction.

Use it forPredicting Java method names from source code; Learning semantic embeddings for code snippets; Researching machine learning models for code understanding

Agent Memory Service

A Node.js web service that provides persistent, structured memory for AI coding assistants and autonomous agents.

ProblemAI assistants typically lose context between conversations, forcing users to re-explain project details or causing agents to make inconsistent decisions due to lack of long-term memory.

Use it forProviding persistent context to Claude Code or Cursor to maintain project state across sessions; Tracking technical decisions and 'gotchas' for autonomous coding agents; Building a long-term memory layer for custom LLM applications using a REST API

ndpvt-web/capy-cortex

Autonomous memory and learning system for AI coding agents.

AgentTrust

AgentTrust is a Python framework that acts as a deterministic safety layer for tool-using AI agents.

ProblemAI agents can cause irreversible damage through common, enumerable operations (e.g., rm -rf, credential exfiltration) that are difficult to detect via intent analysis alone.

Use it forSecuring autonomous coding agents against accidental or malicious destructive commands; Enforcing least-privilege access controls in multi-agent systems; Auditing and logging agent tool usage for compliance

DDPM + IPA Protein Generation

A PyTorch implementation of a denoising diffusion probabilistic model (DDPM) combined with invariant point attention (IPA) for generating protein structures and sequences.

ProblemTraditional protein design methods often struggle with generating valid, stable 3D structures that satisfy physical constraints; this approach uses diffusion and equivariant attention to better model structural geometry.

Use it forGenerating novel protein structures from scratch; Infilling missing regions of known protein structures; Researching equivariant diffusion models for molecular biology

LLM Structured Output Benchmarks

A benchmarking suite that evaluates various LLM structured output frameworks (such as Instructor, Mirascope, and LangChain) on tasks like multi-label classification, named entity recognition, and synthetic data generatio.

ProblemDevelopers lack objective, comparative data on the performance, reliability, and latency of different LLM structured output frameworks, making it difficult to choose the right tool for their specific task.

Use it forComparing the reliability of different structured output libraries before integration; Evaluating the latency impact of constrained decoding methods like Outlines or LMFormatEnforcer; Assessing the quality of named entity recognition results across different frameworks

outlines-core

A Rust library for structured generation that builds regular expressions from JSON schemas and constructs finite-state automata to map vocabulary tokens to state transitions.

ProblemEnsuring LLMs generate outputs that strictly adhere to specific data structures or grammars without post-processing or validation failures.

Use it forConstraining LLM outputs to specific JSON formats; Generating valid code or structured data using finite-state automata; Integrating schema-constrained generation into Python LLM applications

py-agent-framework

A Python library built on pydantic-ai that adds stateful session persistence, automatic context-window management, and a lifecycle extension system to LLM agents.

ProblemPydantic AI lacks built-in support for multi-turn conversation persistence and automatic context window management, forcing developers to manually handle history storage and truncation logic.

Use it forBuilding multi-turn conversational agents that require memory across sessions; Managing long-running agent interactions that exceed model context windows; Extending pydantic-ai agents with custom lifecycle hooks for tools and skills

alexpota/pg-agent-memory

Stateful AI agent memory layer for PostgreSQL with pgvector - TypeScript-first with intelligent context management.

AGDe-Judge

This is the official code implementation for the EMNLP 2025 paper on Assistant-Guided Mitigation of Teacher Preference Bias in LLM-as-a-Judge.

ProblemLLM-as-a-Judge systems often inherit or amplify the preference biases of the teacher model used to generate training data, leading to unfair or skewed evaluations.

Use it forTraining LLM judges to reduce preference bias towards specific teacher models; Generating high-quality DPO and SFT datasets for evaluation tasks; Benchmarking judge models on bias metrics like OffsetBias and MT-Bench

Judging the Judges: LLM-as-a-Judge Bias Evaluation

A research framework for systematically evaluating and mitigating biases in LLM-as-a-Judge pipelines.

ProblemLLM judges exhibit systematic biases (position, verbosity, self-preference) that compromise the validity of automated evaluations, and there is no clear guidance on which mitigation strategies work best or how they inter

Use it forEvaluating the reliability of LLM judges for automated benchmarking; Selecting appropriate debiasing strategies for specific evaluation tasks; Analyzing the cost-accuracy tradeoff of different ensemble methods

Self-Preference Bias in LLM Juries: A Controlled Test of Hard Judge Exclusion

This is a research paper and accompanying code repository that empirically tests whether excluding the generating model from its own LLM judge ensemble (hard holdout) reduces self-preference bias.

ProblemSelf-preference bias in LLM-as-a-Judge ensembles where the model being evaluated votes for itself, leading to skewed evaluation results.

Use it forDesigning LLM-as-a-Judge evaluation pipelines; Mitigating self-preference bias in automated model comparisons; Reproducing statistical analyses of LLM jury dynamics

kaist-cvml/perception-judge

[ICML 2026] Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling.

AI Agent Temporal Memory Management System

A database-backed system for LLM agents that uses PostgreSQL and pgvector to store, retrieve, and version long-term memories.

ProblemLLM agents lack a robust mechanism to store, prioritize, and update long-term memories, leading to context loss or reliance on outdated information.

Use it forManaging long-term context for autonomous AI agents; Resolving conflicting memories through temporal versioning; Performing semantic search over agent interaction history

Dynamic RAG Knowledge Updater

A production-grade Retrieval-Augmented Generation (RAG) system that implements semantic corpus versioning and hybrid retrieval.

ProblemStandard RAG systems often suffer from stale knowledge, lack of version control for documents, and high rates of hallucination due to unverified retrieval results.

Use it forBuilding enterprise search engines that require version control over document corpora; Implementing RAG pipelines with automatic hallucination mitigation; Managing dynamic knowledge bases where documents are frequently updated or replaced

Awesome LLM Agent Trajectory Analysis

A curated repository and survey companion for LLM agent trajectory analysis, accepted by IEEE TSE 2026.

ProblemThe rapid growth of LLM agent systems has led to a fragmented and disorganized body of research on how to diagnose, debug, and fix their non-deterministic failures.

Use it forLiterature review for researchers studying LLM agent failures; Finding benchmarks and datasets for agent debugging; Identifying state-of-the-art methods for root cause analysis in agent trajectories

Agents Failure Attribution Benchmark

This repository provides the implementation and dataset for the ICML 2025 Spotlight paper 'Which Agent Causes Task Failures and When?'.

ProblemManual inspection of failure logs in multi-agent systems is labor-intensive and difficult due to the complexity of interactions between agents.

Use it forDebugging complex multi-agent system failures automatically; Evaluating the effectiveness of different failure attribution methods (e.g., step-by-step vs. all-at-once); Providing intermediate feedback signals for agent self-improvement or reinforcement learning

agent-belt

agent-belt is a CLI tool for reproducible evaluation of AI coding agents.

ProblemExisting evaluation tools typically test model outputs or internal agent loops, but do not evaluate the specific CLI binaries that end-users install and run, leaving a gap in verifying the actual user-facing agent behavi

Use it forRegression testing for AI coding agent integrations; Comparing performance across different coding agent CLIs; Measuring the reliability and variance of stochastic agent outputs

OWASP Agent Security Regression Harness

A vendor-neutral CLI tool for running executable security regression tests against agentic applications and MCP-integrated systems.

ProblemLack of executable, reproducible security regression testing for agentic applications and MCP systems, making it difficult to verify that changes do not reintroduce known security failures.

Use it forVerifying that prompt or model updates do not reintroduce known agent security vulnerabilities; Testing MCP integrations for unauthorized tool calls or data exfiltration; Generating machine-readable security test results for CI/CD pipelines

SkillForge: Compositional Skill Synthesis for Verified Dafny Programs

SkillForge is a framework that generates formally verified Dafny programs from natural language by decomposing the task into a library of atomic, reusable skills.

ProblemExisting approaches to generating verified code either fail without recourse when verification fails or rely on non-deterministic, opaque agentic reasoning that is difficult to debug and control.

Use it forGenerating formally verified code from natural language specifications; Automating the repair of formal verification failures in Dafny; Benchmarking compositional LLM agent architectures for code synthesis

Predicting Program Exit Code with LLMs and Programming Language Semantics

Large language models (LLMs) have shown proficiency in various software engineering tasks, such as code generation and translation.

Relational Task Generation Language (RTGL)

RTGL is a declarative specification framework for defining prediction tasks in Relational Deep Learning (RDL).

ProblemManually defining RDL prediction tasks is laborious and frequently results in data leakage due to inconsistent, hand-crafted SQL definitions.

Use it forDefining new Relational Deep Learning benchmark tasks without manual SQL; Reconstructing and validating existing RDL benchmarks for consistency; Designing diverse RDL tasks with different target types and forms

Self-Reports Are Not Verification: Environment-Grounded Auditing of LLM Operators in Evolutionary Search

This paper introduces an environment-grounded audit framework to verify the reliability of LLM agent self-reports during evolutionary search.

ProblemLLM agents' self-reported confidence and rationales are often used as monitoring signals but are not verified against actual outcomes, leading to potential misjudgment of agent performance.

Use it forAuditing the reliability of LLM agent self-reports in automated search tasks; Evaluating the calibration of LLM confidence scores in evolutionary algorithms; Designing verification mechanisms for autonomous agent actions

ContextPipe: Database-Inspired Context Assembly for Long-Horizon Agents

ContextPipe is a framework that treats LLM context assembly as a database query execution problem, using a five-phase pipeline (Plan, Bind, Optimize, Execute, Feedback) to manage prompt history.

ProblemContext assembly logic in production agentic systems is scattered, ad hoc, and difficult to audit or optimize under hard context-window and cache constraints.

Use it forManaging context windows for long-horizon software engineering agents; Optimizing token usage and LLM call frequency in agentic workflows; Debugging and auditing prompt construction in production LLM systems

Dual Process Motion Planning

This paper introduces a neuro-symbolic framework for nonlinear motion planning that combines fast, experience-driven learning modules with robust symbolic solvers.

ProblemClassical planning methods are computationally expensive and lack adaptability, while pure learning-based approaches often lack the robustness and precise guarantees required for reliable robotic operation.

Use it forReal-time robotic navigation in complex environments; Adaptive motion planning for industrial automation; Generalizing planning strategies across different robotic tasks

CARE: Contrastive Anchor-based Rubric Evolution for LLM Post-Training

CARE is a method for rubric-based reinforcement learning that dynamically evolves evaluation rubrics by contrasting model rollouts against high-quality anchor responses.

ProblemStatic rubrics in reinforcement learning are easily hacked as the policy evolves, and existing dynamic rubric methods suffer from undirected extraction, unreliable hack detection, and unbounded rubric proliferation.

Use it forPost-training LLMs on open-ended instruction following tasks; Preventing reward hacking in rubric-based RL; Improving model performance on benchmarks like Arena-Hard-2.0 and FollowBench

Darkbloom: Private Inference Network on Idle Macs

Darkbloom is a decentralized network that utilizes idle Apple Silicon Macs to provide private, OpenAI-compatible LLM inference.

ProblemCentralized AI inference providers have full access to user data, and existing local solutions lack the privacy guarantees needed when inference is run on hardware owned by a third party.

Use it forRunning large language models locally on Apple Silicon hardware while maintaining privacy from the machine owner; Monetizing idle Apple Macs by providing inference services to a decentralized network; Deploying OpenAI-compatible AI applications with enhanced privacy guarantees via hardware-encrypted enclaves

FinLifeBench

FinLifeBench is a benchmark for evaluating large language models on longitudinal banking dialogue tasks.

ProblemExisting benchmarks fail to test the exhaustive, longitudinal reconstruction of customer states and life events over many sessions, which is critical for real-world banking assistants.

Use it forEvaluating LLM performance in maintaining long-term customer records; Benchmarking models on longitudinal state reconstruction in financial services; Assessing the impact of context length on event recall and state accuracy

Denoising Diffusion Generative Models Secretly Calculate Attentions

This paper demonstrates that denoising diffusion models inherently utilize an attention mechanism similar to that found in transformers.

ProblemDenoising diffusion models are computationally intensive and require lengthy training processes compared to transformer-based approaches.

Use it forDeveloping more efficient image generation models by replacing diffusion steps with attention mechanisms; Analyzing the theoretical equivalence between diffusion models and transformer architectures; Optimizing computational costs for generative AI training and inference

Evret

Evret is a lightweight Python framework for evaluating information retrieval systems, providing standard IR metrics like NDCG, MRR, and Recall.

ProblemLack of a lightweight, focused library for standard Information Retrieval metrics in modern AI and RAG stacks.

Use it forEvaluating RAG pipelines; Benchmarking vector database search quality; Testing retrieval systems with custom LLM judges

mcp-server-tester

A Playwright-based testing and evaluation framework for MCP servers that allows developers to write deterministic unit tests and run data-driven evals using LLM-as-a-judge.

ProblemThe lack of a standardized, easy-to-use testing framework for Model Context Protocol (MCP) servers, which currently makes it difficult to write deterministic tests for tool calls and evaluate LLM-driven tool usage.

Use it forRegression testing MCP server tools in CI pipelines; Validating MCP protocol conformance and tool schemas; Evaluating LLM tool discoverability and response quality using rubrics

Bedrock Knowledge Base Drift Detection

A Python framework for measuring semantic drift and quality degradation in Amazon Bedrock RAG pipelines.

ProblemLack of quantitative methods to verify that swapping foundation models in RAG systems does not degrade response quality or factual grounding.

Use it forValidating model upgrades in production RAG systems; Monitoring retrieval quality degradation over time; Automating regression testing for LLM-based applications

AVS Framework

An open-source Python library for auditing AI-driven employment decision systems.

ProblemThe lack of independent, accessible tools to verify that AI hiring systems comply with anti-discrimination laws, especially when vendors refuse to share source code or audit results.

Use it forAuditing applicant tracking systems (ATS) for discriminatory selection rates across protected groups; Compliance reporting for NYC Local Law 144 and EEOC guidelines; Detecting model drift in automated hiring tools over time

ailedger-detection

A Python library providing statistical primitives for detecting bias and drift in AI decision events.

ProblemLack of standardized, auditable statistical tools for detecting bias and drift in AI decision-making that align with specific regulatory frameworks like EEOC and FDIC.

Use it forAuditing AI hiring models for disparate impact against EEOC four-fifths rule; Monitoring credit decision models for population stability drift per FDIC guidelines; Verifying statistical parity in automated decision systems for regulatory compliance

Pramana API

An open-source backend service for crowdsourced LLM drift detection that tracks model output consistency over time.

ProblemLLM providers silently update models, causing the same prompts to return different or degraded results over time without explicit notification.

Use it forMonitoring LLM provider model updates for silent behavior changes; Tracking output consistency of specific prompts across different model versions; Crowdsourced benchmarking of LLM stability

drift-paper-experiments

A repository containing data and code for experiments that replicate and critique the findings of the paper 'Open-Source Drift Detection Tools in Action: Insights from Two Use Cases'.

ProblemLack of clear, practical evidence on whether detected data drift actually leads to significant model performance degradation in specific use cases.

Use it forReplicating and extending drift detection experiments from academic literature; Evaluating the practical impact of data drift on model accuracy; Comparing different drift detection tools in a controlled environment

GraphWiz: Instruction-Following LLM for Graph Problems

This repository provides the code, data, and models for GraphWiz, a series of instruction-tuned large language models designed to solve graph theory problems.

ProblemStandard LLMs struggle with explicit graph reasoning and often fail to provide transparent reasoning paths for structural problems, limiting their reliability in graph-centric applications.

Use it forSolving graph theory problems like cycle detection and shortest path using natural language prompts; Training custom LLMs for specific graph reasoning tasks using the provided GraphInstruct dataset; Benchmarking LLM performance on complex graph structures against baselines like GPT-4 and Mistral

MedReason: Medical Reasoning Dataset and Models

MedReason is a large-scale dataset of 32,682 medical question-answer pairs with step-by-step reasoning chains generated using a medical knowledge graph.

ProblemLLMs often produce unfaithful or hallucinated medical reasoning; MedReason addresses this by providing high-quality, knowledge-graph-grounded reasoning steps for training and evaluation.

Use it forFine-tuning LLMs for medical question answering; Generating explainable medical reasoning chains; Evaluating LLM performance on medical benchmarks like MedQA and Medbullets

Judge Reliability Harness

An open-source framework for generating validation suites to evaluate the reliability of LLM-as-a-Judge systems.

ProblemLLM judges are widely used but their reliability (invariance to formatting, paraphrasing, and sampling noise) is rarely systematically evaluated or reported, making it difficult to trust AI evaluation results.

Use it forValidating the robustness of LLM judges against input perturbations like paraphrasing or verbosity; Generating standardized reliability reports for AI evaluation benchmarks; Comparing the performance and cost of different LLM judge configurations

acceptance-bench

A Python-based LLM evaluation framework that measures model acceptance and refusal rates across creative writing tasks.

ProblemStandard LLM benchmarks often only measure binary refusal rates, failing to capture 'soft refusals' (hedging, disclaimers) or the impact of prompt variation and temperature on output quality.

Use it forBenchmarking LLMs for creative writing capabilities; Comparing refusal policies across different AI models; Evaluating model sensitivity to prompt phrasing and temperature settings

DSPy Micro Agent

A minimal agent runtime built on DSPy modules that implements Plan/Act/Finalize loops with OpenAI and Ollama support.

ProblemSimplifies the implementation of tool-using agents by abstracting away complex orchestration logic into a thin, readable Python loop and standard DSPy signatures.

Use it forLocal LLM experimentation with tool-calling capabilities; Serving lightweight AI agents via HTTP API; Evaluating small language model performance on simple tasks

SynSem-PatternBank

A repository of syntactic and semantic patterns for Japanese text.

ProblemLack of reproducible and deduplicated syntactic/semantic pattern resources for Japanese NLP research.

Use it forReproducible Japanese NLP research; Structural deduplication of text patterns; Semantic pattern analysis

LossBench

LossBench is a framework for evaluating and controlling financial agents using severity-weighted expected loss rather than simple accuracy.

ProblemStandard accuracy metrics fail to capture the high cost of errors in financial operations, leading to unsafe agent deployments where rare but severe errors are overlooked.

Use it forEvaluating LLM agents on financial reconciliation and payment repair tasks; Simulating the impact of different risk policies on historical agent workloads; Monitoring and auditing agent decisions in production financial systems

causal-agent-replay

A Python library for counterfactual replay and causal attribution over LLM agent trajectories.

ProblemExisting observability tools show what happened, and eval tools score pass/fail, but neither identifies which specific step in an agent's trajectory actually caused a bad outcome.

Use it forDebugging LLM agents that issue incorrect refunds or leak data by identifying the specific decision step that caused the; Evaluating the impact of model upgrades or prompt changes on agent outcomes via policy and context interventions; Generating interactive attribution reports with confidence intervals for agent failures in LangGraph, OpenAI Agents SDK,

DVAIA - Damn Vulnerable AI Application

DVAIA is an open-source web application designed for LLM red teaming and AI security training.

ProblemLack of a dedicated, easy-to-deploy, and comprehensive platform for hands-on training in LLM-specific vulnerabilities like prompt injection and RAG poisoning.

Use it forTraining security professionals in LLM red teaming techniques; Testing the robustness of RAG pipelines against vector store poisoning; Practicing indirect prompt injection via document and audio files

AInfer-PD: Communication-Safe In-Place Prefill-Decode Multiplexing

This paper proposes AInfer-PD, a system design for distributed Mixture-of-Experts (MoE) inference that safely multiplexes prefill and decode phases on shared accelerators.

ProblemPersistent coexistence of prefill and decode phases in agentic RL causes latency interference and communication conflicts in distributed MoE deployments, leading to prolonged rollout completion times.

Use it forAccelerating rollout inference for large-scale agentic reinforcement learning; Optimizing distributed MoE model serving with mixed prefill and decode workloads; Reducing wall-clock time for multi-turn LLM agent trajectories

CoSMO: Goal-Oriented Status Updating and Selective Offloading

This paper proposes CoSMO, a cooperative event-driven reinforcement learning framework for edge-cloud computing.

ProblemTraditional freshness-driven schemes (like Age of Information) do not directly value status updates by their effect on subsequent task decisions, leading to suboptimal offloading choices under partial observability.

Use it forOptimizing task offloading decisions in edge computing environments with intermittent cloud connectivity; Improving on-time completion rates for latency-sensitive applications in distributed systems

mzCache: On-Device LLM Memory Management under Multitasking

mzCache is an on-device LLM inference system designed for mobile multitasking environments.

ProblemMobile operating systems evict LLM memory (weights and KV cache) under multitasking pressure, forcing slow storage reads or full KV cache recomputation that severely degrades inference responsiveness.

Use it forRunning LLMs on mobile devices while users switch between apps; Reducing latency spikes caused by OS memory pressure during multitasking; Optimizing Time-to-First-Token for on-device AI assistants

Contribution-Aware Bandwidth Allocation for Multimodal Split Learning

This paper introduces ModalShare, a method for optimizing bandwidth usage in multimodal split learning by allocating compression budgets based on each modality's contribution to the final prediction.

ProblemExisting compression schemes for multimodal split learning allocate bandwidth equally or based on data dimension, ignoring the actual predictive contribution of each modality, which leads to suboptimal model accuracy und

Use it forTraining multimodal AI models on edge devices with limited uplink bandwidth; Optimizing data transmission in federated or split learning systems with multiple sensor inputs; Improving model accuracy in resource-constrained environments where equal compression ratios are suboptimal

Just Talk Once: Communication-Efficient Split Federated LLM Fine-Tuning on Edge Devices

This paper proposes L-shaped SFT and one-shot SFT, frameworks for split federated fine-tuning of LLMs on edge devices.

ProblemConventional split federated fine-tuning requires continuous bidirectional communication and client involvement, which is unsustainable for edge devices with limited bandwidth and connectivity.

Use it forFine-tuning LLMs on commercial smartphones with limited battery and connectivity; Federated learning scenarios where clients have intermittent network access; Reducing communication overhead in split federated learning systems

OptiTrain: Fault-Tolerant Distributed LLM Training

OptiTrain is a distributed training and serving engine designed for ephemeral GPU clusters, featuring elastic scaling and rapid fault recovery.

ProblemStandard distributed training frameworks fail catastrophically when ephemeral GPU instances crash, requiring manual restarts and causing significant downtime and resource waste.

Use it forTraining LLMs on spot or ephemeral GPU instances where node failure is frequent; Serving LLMs with high throughput and low latency using dynamic batching and KV caching; Building resilient ML infrastructure that can survive multi-node failures without manual intervention

Prometheus-Vision

Prometheus-Vision is an open-source Vision-Language Model (VLM) designed to act as an evaluator for fine-grained scoring.

ProblemExisting evaluation methods for Vision-Language Models are either too expensive (GPT-4V), subjective (human evaluation), or lack the granularity to assess specific customized rubrics.

Use it forAutomated evaluation of multimodal AI responses against specific quality criteria; Benchmarking Vision-Language Models using the Perception-Bench dataset; Replacing expensive proprietary LLM judges with a local, open-source model

Prometheus: Inducing Fine-grained Evaluation Capability in Language Models

Prometheus is an open-source evaluator language model designed for fine-grained evaluation of LLM outputs using customized score rubrics.

ProblemThe lack of open-source, reproducible, and cost-effective tools for fine-grained evaluation of LLM outputs, which are currently dominated by expensive proprietary models (like GPT-4) or subjective human evaluation.

Use it forAutomated evaluation of LLM responses against specific quality criteria; Benchmarking LLM performance on tasks like Vicuna or MT-Bench; Fine-tuning custom evaluator models using the provided Feedback Collection dataset

openJiuwen Memory

A Python library providing long-term memory capabilities for AI agents running on the openJiuwen framework.

ProblemStandard LLM agents lack persistent long-term memory, requiring manual implementation of storage, retrieval, and extraction logic for conversation history and user preferences.

Use it forAdding persistent memory to LLM agents built with the openJiuwen framework; Managing user profiles and conversation histories with semantic search; Migrating agent memory data between different storage backends

Agent Design Patterns

A 7x6 framework for agent architecture that organizes 28 design patterns by cognitive function and execution topology.

ProblemMost agent architecture guides provide flat lists of patterns that fail to explain where a specific problem sits in the design space or which pattern is appropriate for a specific cognitive/execution coordinate.

Use it forSelecting the appropriate agent pattern for a specific architectural coordinate; Implementing specific agent behaviors like RAG, Chain of Thought, or Multi-Agent collaboration; Studying the structural differences between flat agent lists and matrix-based design spaces

Rubricon: Evaluation-First Attention for LLMs

Rubricon is a Python library that implements the Evaluation-First Attention (EFA) paradigm, which inverts standard LLM workflows by generating evaluation criteria before text generation.

ProblemStandard LLM pipelines generate text without awareness of quality dimensions, leading to holistic but untargeted refinement feedback and equal emphasis on all criteria regardless of failure severity.

Use it forGenerating high-quality technical documentation with guaranteed adherence to specific style and accuracy rubrics; Creating educational content that meets predefined pedagogical standards; Automated quality assurance for LLM outputs in production pipelines

mlx-serve

A native Zig-based inference server for Apple Silicon that runs MLX and GGUF models without Python.

ProblemExisting local LLM tools like Ollama or LM Studio often lack full API compatibility, have slower performance on Apple Silicon, or require Python environments; mlx-serve addresses this with a native Zig implementation tha

Use it forRunning local LLMs on Apple Silicon with OpenAI/Anthropic API compatibility; Replacing Ollama or LM Studio for local inference with faster decode speeds; Generating images, video, music, and speech locally via a unified server

Lucebox: LLM Speculative Inference Server

Lucebox is a C++17 inference engine optimized for heterogeneous hardware and consumer GPUs, utilizing custom kernels and speculative decoding techniques like DFlash and KVFlash.

ProblemStandard inference engines often fail to fully utilize consumer GPU capabilities or lack optimized speculative decoding paths for heterogeneous and non-datacenter hardware.

Use it forAccelerating LLM inference on consumer-grade NVIDIA GPUs (e.g., RTX 3090); Optimizing inference for AMD APUs and heterogeneous CPU/GPU setups; Deploying large language models like Qwen, Gemma, and DeepSeek on local hardware

StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts

This repository provides the dataset and PyTorch code for the AAAI 2022 paper on StepGame, a benchmark for evaluating multi-hop spatial reasoning in text.

ProblemLack of a robust, large-scale benchmark specifically designed to test multi-hop spatial reasoning capabilities in natural language processing models.

Use it forBenchmarking NLP models on spatial reasoning tasks; Training and evaluating models for multi-hop question answering; Researching robustness of language models to noisy spatial descriptions

ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs

This is the official PyTorch implementation of the ConE model, a knowledge graph embedding method that represents entities as cones in a hyperbolic space to support multi-hop reasoning.

ProblemStandard Euclidean embeddings struggle to represent hierarchical and tree-like structures in knowledge graphs, leading to poor performance in multi-hop reasoning tasks.

Use it forMulti-hop reasoning over knowledge graphs; Knowledge graph completion and link prediction; Representing hierarchical data in hyperbolic space

THU-KEG/DacKGR

Source codes and datasets for EMNLP 2020 paper "Dynamic Anticipation and Completion for Multi-Hop Reasoning over Sparse Knowledge Graph".

agents-flex

A lightweight Java framework for building AI agents, positioned as a counterpart to Spring AI.

ProblemLack of a lightweight, dedicated Java framework for AI agent development comparable to Spring AI.

Use it forBuilding Java-based AI agents with RAG capabilities; Integrating Model Context Protocol (MCP) servers into Java applications; Developing agents with text-to-speech and speech-to-text features

mathomhaus/guild

Shared context, memory, and task coordination across AI coding agents.

shinpr/mcp-local-rag

Local-first RAG server for developers.

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

This paper introduces BERT, a language representation model that pre-trains deep bidirectional representations from unlabeled text by conditioning on both left and right context.

ProblemPrevious language representation models were unidirectional or could not be easily adapted to diverse downstream NLP tasks without substantial architecture modifications.

Use it forQuestion answering systems; Natural Language Inference; Text classification

Agentic Memory

A research notebook implementing cognitive architecture concepts for LLM agents.

ProblemLLMs are stateless and lack the ability to naturally recall past experiences, learn from them, or maintain long-term context across interactions.

Use it forBuilding multi-turn LLM agents that retain long-term context; Implementing psychological memory models in AI systems; Educational reference for cognitive architectures in LLMs

7/24 Office

A self-evolving AI agent system built in pure Python with a modular architecture, featuring 36 tools, three-layer memory, and MCP plugin support.

ProblemAddresses the lack of lightweight, framework-agnostic AI agent systems that can run 24/7, self-repair, and dynamically extend their own toolset without relying on heavy orchestration libraries like LangChain or CrewAI.

Use it forPersonal AI assistant running 24/7 via messaging platforms; Automated task scheduling and execution with persistent state; Multi-tenant AI service deployment with per-user containers

coleam00/claude-memory-compiler

Give Claude Code a memory that evolves with your codebase.

auto-deep-researcher-24x7

An autonomous AI agent that manages deep learning experiments 24/7 using a Leader-Worker architecture.

ProblemDeep learning experiments require constant monitoring and manual intervention, preventing researchers from running experiments continuously or efficiently managing long-running jobs.

Use it forAutomating hyperparameter tuning and model training overnight; Running experiments on remote Slurm clusters or SSH servers; Tracking experiment history and insights to avoid repeating failed approaches

CIPR: Benchmark for Coding Agent Vulnerability to Repository Poisoning

This paper introduces CIPR, a benchmark of 1,920 instances designed to evaluate how user-side prompt configurations (PLCs) affect coding agents' susceptibility to repository poisoning.

ProblemPrior research on repository poisoning focused primarily on attacker-side injection techniques, neglecting the significant role that everyday user invocation choices (task type, prompt phrasing, and supplied skills) play

Use it forEvaluating the security robustness of LLM-based coding agents against prompt injection and repository poisoning.; Analyzing how specific user prompt phrasing and task delegation choices influence the likelihood of successful attacks.; Benchmarking the effectiveness of agent alert mechanisms under different contextual configurations.

SingProbe Technical Report

SingProbe is a lightweight runtime guard for LLMs that reuses hidden states during inference to predict query intent, safety, and hallucination risk at the token level.

ProblemExisting runtime guardrails rely on independent external models, introducing additional inference cost, delayed safety signals, and capacity mismatch with increasingly capable base models.

Use it forReal-time safety monitoring during LLM text generation; Detecting hallucinations in LLM outputs without external models; Guiding constrained safe decoding in medical LLM applications

The Fragility of Jailbreak Robustness Across Operational States

This paper investigates how the robustness of LLMs against jailbreak attacks changes when the system prompt is modified, even if the prompt is not designed to affect safety.

ProblemStandard jailbreak evaluations rely on a single 'vanilla' configuration, failing to capture how robustness degrades or changes when ordinary system prompts are altered, leading to an incomplete picture of model safety.

Use it forDesigning more comprehensive safety evaluation suites for LLMs; Understanding the sensitivity of model alignment to system prompt variations; Developing robust safety mechanisms that account for diverse operational contexts

DP-VOXLET: Provable Speaker Anonymization

This paper introduces a formal definition of speaker differential privacy and a mechanism for speaker anonymization that provably satisfies this definition.

ProblemExisting speaker anonymization systems rely on heuristic methods that lack formal privacy guarantees, leaving them vulnerable to sophisticated re-identification attacks without a provable lower bound on success rates.

Use it forAnonymizing speech data for privacy-preserving machine learning training; Creating synthetic speech datasets with guaranteed privacy bounds; Compliance with privacy regulations requiring provable anonymization guarantees

Auditing Anonymous AI Models: A Four-Stage Protocol for Black-Box Identity Verification

This paper proposes a forensic audit protocol to verify the identity of anonymous AI models served via API.

ProblemThe lack of a validated, accurate methodology for black-box identity verification of anonymous AI models, where self-identification is untrustworthy and existing checklists lack accuracy evidence.

Use it forVerifying the identity of stealth-released frontier models; Assessing supply-chain risks for API-based LLM integrations; Detecting configuration drift between preview and production model versions

flake16-framework

This is a replication package for the ICST 2022 paper on detecting flaky tests using machine learning.

ProblemLack of public, reproducible data and code for evaluating machine learning features in the context of flaky test detection.

Use it forReproducing the results of the ICST 2022 paper on flaky test detection; Researching feature engineering for machine learning models applied to software testing; Analyzing datasets of flaky test executions

GRAIN: Bridging Name and Narrative Shifts in Real-World Graph Reasoning

GRAIN is a single-agent framework that uses reinforcement learning to optimize LLMs for robust graph reasoning.

ProblemLLMs are brittle to shifts in node identifiers and task formulation, often overfitting to surface patterns, while multi-agent solutions to mitigate this incur prohibitive latency.

Use it forExtracting topological structures from noisy, real-world text descriptions; Performing graph reasoning tasks where node identifiers or task phrasing vary; Reducing latency in multi-agent graph processing pipelines

Automated Testing of LLM-Based Post Hoc Explainers Using Model Checking

This paper proposes a method to systematically test the faithfulness of natural language explanations generated by LLMs for sequential decision-making policies.

ProblemLLMs often generate plausible but incorrect explanations for policy decisions, and there is no existing systematic approach to test whether these explanations are faithful to the underlying environment due to the lack of

Use it forEvaluating the reliability of LLM-generated explanations for reinforcement learning policies; Benchmarking open-weight LLMs on their ability to accurately describe MDP environments; Identifying specific types of policy behavior that LLMs struggle to explain correctly

Timing-Aware Repurchase Prediction for Web-Scale E-Commerce

This paper proposes using survival models, specifically Accelerated Failure Time (AFT) models, to predict the time-to-repurchase for grocery items instead of using multiple binary classifiers for different time horizons.

ProblemCurrent repurchase recommenders require training and maintaining a separate binary classifier for every specific time horizon (e.g., 7 days, 30 days), leading to model sprawl and inefficiency.

Use it forGrocery e-commerce recommendation systems; Predicting customer repurchase timing for inventory planning; Replacing multiple binary classifiers with a unified survival model

Cognitive Capability Profiling for AI Workplace Suitability

This paper introduces a pipeline that profiles AI agents and workplace tasks using a shared set of core cognitive capabilities.

ProblemAggregate benchmark scores fail to predict real-world task success, and human judgments of AI capabilities become outdated quickly, leaving organizations without a reliable way to scope AI for specific jobs.

Use it forIdentifying which workplace tasks are suitable for AI automation; Scoping AI pilot programs within an organization; Comparing AI systems based on specific cognitive strengths rather than aggregate scores

DUOTRACE: Cascade Failure Attribution for Multi-Agent Systems

DUOTRACE is a detection filter designed to improve failure attribution in LLM-based multi-agent systems.

ProblemExisting failure attribution methods either overlook fine-grained semantics (topology/spectrum-based) or suffer from long-context degradation when processing lengthy agent trajectories (LLM-based).

Use it forDebugging complex multi-agent workflows where long trajectories cause LLM context degradation; Improving the accuracy of automated failure attribution in LLM agent pipelines; Filtering noisy agent trajectories to provide focused evidence for downstream analysis

PrismF: Multi-Perspective Enhancement and Progressive Fusion for Multimodal Entity Representation Learning

PrismF is a framework for multimodal knowledge graph completion that addresses semantic over-smoothing and noise in entity representations.

ProblemExisting methods suffer from semantic over-smoothing within modalities and ineffective noise filtration across modalities, particularly under sparse or ambiguous conditions.

Use it forMultimodal knowledge graph completion; Entity representation learning in sparse or ambiguous conditions; Reasoning tasks involving diverse multimodal inputs

DoCtOR: Automated Failure Attribution for Multi-Agent Systems

DoCtOR is a reflection framework for LLM-based multi-agent systems that identifies the specific agent responsible for a failure rather than forcing all agents to reflect.

ProblemCurrent self-reflection methods in multi-agent systems force all agents to reflect on failures, which contaminates the memory of agents that behaved correctly with incorrect insights, leading to degraded performance.

Use it forImproving success rates in multi-agent systems for complex reasoning tasks like HotPotQA; Enhancing performance in data extraction and chart understanding tasks like ChartQAPro; Optimizing web navigation and interaction agents in benchmarks like Mind2Web

When Robots Mishear Us: Mapping the Safety Risks of Voice-Controlled Embodied AI

This paper investigates how automatic speech recognition (ASR) errors in voice commands can compromise the safety of Embodied AI (EAI) models.

ProblemVoice-controlled Embodied AI models may execute unsafe or harmful actions when user speech is misrecognized by ASR systems, as current safety benchmarks do not account for ASR-induced errors.

Use it forEvaluating the robustness of voice-controlled robot safety mechanisms; Identifying specific types of ASR errors that bypass AI refusal behaviors; Developing better error-correction strategies for voice interfaces in embodied AI

BLOOM-WILT: Logit Tilting for Behaviour Elicitation

BLOOM-WILT is an automated auditing pipeline that uses logit tilting to elicit rare, unsafe behaviors from large language models.

ProblemStandard automated auditors are sample-inefficient and fail to surface rare behaviors that only emerge after extensive real-world deployment interactions.

Use it forSafety auditing of deployed LLMs; Eliciting rare self-harm or toxic behaviors for evaluation; Benchmarking model safety rankings

HiRS-Agent: A Hierarchical Multi-Agent System for Reliable Long-Horizon Remote Sensing Task Solving

HiRS-Agent is a hierarchical multi-agent system designed to solve complex, long-horizon remote sensing tasks by separating high-level planning from low-level execution.

ProblemExisting monolithic agentic frameworks for remote sensing suffer from unstable execution, incorrect tool usage, and error propagation due to centralized decision-making that fails to handle multi-stage interdependencies.

Use it forAutomating complex remote sensing workflows that require multiple interdependent steps; Improving the reliability of LLM-based agents in geospatial data processing; Reducing error propagation in multi-stage remote sensing analysis pipelines

Agent2UCB: Agentic System for Generative Engine Optimization

Agent2UCB is an agentic system designed to optimize content for visibility in LLM-driven search engines (Generative Engine Optimization).

ProblemThe lack of automated, feedback-driven methods to optimize content for citation by large language models, which is a new and distinct challenge from traditional SEO.

Use it forImproving the likelihood of content being cited by AI search engines like Perplexity or Google AI Overviews; Automating the selection of content optimization strategies based on performance feedback; Monitoring content for readability and EEAT credibility during GEO optimization

Cross-Relational Preference Learning for Better LLM Instruction Following

This paper proposes Cross-Relational Preference Learning (CRPL), a framework for generating preference data that models relationships between different instructions.

ProblemExisting preference learning approaches overlook the relationships between permissible response spaces of different instructions, limiting models' ability to align with subtle and diverse constraint variations.

Use it forImproving LLM performance on complex instruction-following benchmarks; Generating high-quality preference data for Direct Preference Optimization (DPO) or KTO training; Enhancing model alignment with diverse and subtle constraint variations

When Does Bigger Help? A Controlled Study of LLM Scale for Ontology Learning

This paper presents a controlled evaluation of 13 large language models to determine how model scale affects ontology learning performance.

ProblemThe lack of clear understanding regarding how LLM scale impacts specific ontology learning tasks, leading to arbitrary or inefficient model selection.

Use it forSelecting the optimal LLM size for ontology engineering tasks; Benchmarking LLM performance on biomedical and materials science ontologies; Guiding the design of LLM-assisted knowledge graph construction pipelines

Distance-Guided Decoding for Guaranteed Context-Free Grammar Compliance

This paper proposes a lookahead-guided decoding framework for large language models to ensure outputs strictly comply with context-free grammars.

ProblemExisting grammar-constrained decoders often fail to produce complete valid outputs because locally feasible prefixes may not be extendable to acceptance within a finite token budget.

Use it forGenerating syntactically valid JSON structures; Producing valid SQL queries; Generating Linear Temporal Logic (LTL) formulas

CoLa-ICD: A Knowledge-Enhanced Framework for Long-Tail Automated Medical Coding

CoLa-ICD is a machine learning framework designed to improve the accuracy of automated ICD coding, specifically for rare or long-tail diagnoses.

ProblemStandard automated medical coding models struggle with long-tail distributions, where rare ICD codes have limited training data and are easily confused with semantically similar common codes.

Use it forAutomating the assignment of ICD codes in electronic health records; Improving the detection rate of rare diseases in clinical documentation; Enhancing the performance of multi-label classification models in medical NLP

Wan-Streamer Minimal

A static project page documenting a minimal implementation of the Wan-Streamer training mechanism.

ProblemLack of a minimal, documented reference for aligning real-world multi-party meeting streams onto a single causal timeline for joint text, audio, and video prediction.

Use it forDocumenting preliminary validation results for multi-modal causal prediction models; Providing a reference for aligning meeting streams onto a shared causal timeline

Decision Intelligence Runtime (DIR)

DIR is a neuro-symbolic runtime architecture that separates probabilistic LLM reasoning from deterministic execution to ensure safe, auditable AI autonomy.

ProblemCurrent agent frameworks treat LLMs as autonomous executors, leading to non-deterministic behaviors, hallucinations in critical control paths, and a lack of accountability mechanisms in production systems.

Use it forAutomated trading systems requiring strict accountability and capital protection; Multi-agent systems needing to prevent state drift and TOCTOU vulnerabilities; Enterprise AI governance requiring deterministic audit trails for LLM actions

Hermetic Agent

Hermetic Agent is a secure agentic AI runtime built on FastAPI that enforces security through SPIFFE-style identity, an OPA-inspired policy engine, and an HMAC-chained audit trail.

ProblemNaive agent frameworks often grant unrestricted tool access, making them vulnerable to prompt injection attacks that can lead to malicious command execution.

Use it forRunning AI agents in production environments where prompt injection risks must be mitigated; Enforcing attribute-based access control for agent tool usage; Maintaining tamper-evident audit logs for agent actions

mjlab

mjlab is a Python framework that combines the Isaac Lab manager-based API with MuJoCo Warp, a GPU-accelerated physics engine.

ProblemTraditional robot learning frameworks often rely on CPU-bound physics engines or heavy proprietary stacks, limiting the scale of parallel environment simulation and slowing down the training of complex robotic policies.

Use it forTraining humanoid robots like the Unitree G1 for velocity tracking on flat terrain; Performing motion imitation tasks where robots mimic reference motions; Running GPU-accelerated reinforcement learning simulations with thousands of parallel environments

Kogwistar

Kogwistar is a graph-native Python substrate that models knowledge, workflows, and conversation memory as hypergraphs.

ProblemLack of a unified, graph-native foundation for managing AI agent memory, workflow provenance, and security governance in a single, replayable substrate.

Use it forBuilding agent systems with auditable, replayable execution history; Implementing fine-grained access control and governance for AI tools; Creating graph-backed conversation memory with provenance tracking

omlx

omlx is a macOS menu bar application for running LLM inference on Apple Silicon.

ProblemHigh latency and resource inefficiency when running LLMs on Apple Silicon Macs due to lack of optimized batching and caching mechanisms.

Use it forRunning local LLMs on Apple Silicon Macs with reduced latency; Serving OpenAI-compatible APIs for local development; Managing LLM resources via a menu bar interface

mlx-manager

A web application and CLI tool for managing and serving local LLMs on Apple Silicon using the MLX framework.

ProblemThe complexity of juggling multiple tools, config files, and terminal commands to run local LLMs, and the performance overhead of cross-platform abstraction layers like llama.cpp on Apple Silicon.

Use it forRunning local LLMs on Mac with a graphical interface; Serving local models via OpenAI or Anthropic API endpoints; Managing multiple MLX models with LRU eviction and resource monitoring

turboquant-mlx

A Python library for Apple Silicon that implements extreme KV cache compression (1-3 bit) for LLMs using the MLX framework.

ProblemStandard FP16 KV caching consumes excessive memory, limiting context length and model size for local LLM inference on Apple Silicon devices.

Use it forRunning large LLMs like Llama 3 or DeepSeek R1 on Apple Silicon Macs with limited unified memory; Extending context window length for local inference without hitting VRAM limits; Deploying OpenAI-compatible local LLM servers with reduced memory footprint

Cider: W8A8/W4A8 Inference and Optimized SDPA for Apple Silicon

Cider is a Python library built on MLX that enables W8A8 and W4A8 quantized inference on Apple M5+ chips by implementing custom INT8 TensorOps kernels.

ProblemMLX lacks support for W8A8/W4A8 activation quantization and has suboptimal SDPA kernels for grouped-query attention (GQA) decode, limiting inference speed and efficiency on Apple Silicon.

Use it forAccelerating LLM prefill and decode speeds on Apple M5 Macs; Running quantized LLMs with reduced memory footprint on Apple Silicon; Optimizing inference performance for vision-language models (VLMs) like Qwen3-VL

Post-Cortex

Post-Cortex is an MCP server written in Rust that provides persistent, local-first memory for AI assistants.

ProblemAI assistants typically lack persistent memory, losing context between sessions and requiring users to manually re-explain project details or past decisions.

Use it forGiving AI coding assistants like Claude long-term memory of project decisions and conversations; Building local, privacy-preserving semantic search over personal notes or documents; Creating a knowledge graph from unstructured text for Graph-RAG applications

Atmosphere

Atmosphere is a portable AI agent runtime for the JVM that allows a single @Agent class to run across multiple frameworks like Spring AI and LangChain4j.

ProblemLack of a unified, real-time runtime for JVM AI agents that handles streaming, governance, and multi-framework portability in a single library.

Use it forBuilding production-grade AI agents that require real-time streaming to web clients; Implementing governance controls such as human approval gates and cost ceilings for agent actions; Exposing the same agent logic across multiple protocols including MCP, A2A, and chat platforms like Slack and Discord

RAPTOR: Open-Source Multimodal AI Framework & Agent Harness

RAPTOR is an enterprise AI runtime and framework for building agentic applications over multimodal data (video, audio, images, documents).

ProblemThe fragmentation of AI infrastructure, where developers must manually integrate separate tools for search, memory, orchestration, and multimodal processing to build a single agentic application.

Use it forBuilding enterprise search engines over mixed media types (video, audio, text); Creating autonomous agents with persistent memory and tool access via MCP; Implementing hybrid retrieval (BM25, vector, graph) for complex knowledge bases

ClawEnvKit

ClawEnvKit is an open-source Python toolkit for generating and evaluating environments for claw-like agents.

ProblemExisting agent benchmarks rely on manually written tasks and custom verification code, which is time-consuming, expensive, and does not scale to large datasets.

Use it forAutomatically generating large-scale training datasets for agent benchmarks; Evaluating the performance of various agent frameworks (e.g., Claude Code, OpenClaw) against standardized tasks; Creating custom agent evaluation tasks without writing manual test code

AgentCompass

AgentCompass is an open-source evaluation framework for LLM and VLM agents that decouples models, benchmarks, harnesses, and execution environments.

ProblemThe lack of a unified, extensible framework to evaluate LLM agents across diverse benchmarks and execution environments, which often requires custom, non-reproducible scripts for each specific combination of model and ta

Use it forSystematically benchmarking the capabilities of different LLM agents on standardized tasks like SWE-bench; Comparing the performance of various agent harnesses (e.g., Claude Code, OpenHands) across multiple benchmarks; Analyzing agent trajectories and tool calls to identify failure modes and abnormal behavior

HyperDB

HyperDB is a local vector database library designed for LLM agents, featuring a C++ backend optimized with MKL BLAS for hardware acceleration.

ProblemStandard vector databases can be slow or complex to deploy locally for LLM agent workflows.

Use it forLocal semantic search for LLM agents; RAG pipelines requiring low-latency vector retrieval; Indexing and querying document collections with metadata

opencode-mem

A plugin for the OpenCode coding agent that provides persistent, long-term memory using a local Turso/libSQL vector database.

ProblemAI coding agents typically lack persistent memory, losing context between sessions and requiring users to re-explain project details or preferences every time.

Use it forRetaining project-specific technical context across multiple OpenCode sessions; Learning and maintaining a user profile to personalize agent responses; Managing and browsing agent memories via a local web UI

zvec: In-Process Vector Database

zvec is a lightweight, open-source vector database library that runs in-process, eliminating the need for external servers.

ProblemTraditional vector databases require dedicated servers and complex configuration, adding latency and infrastructure costs for applications that only need simple, local, or embedded similarity search capabilities.

Use it forSemantic search in local applications without a database server; Hybrid retrieval combining vector similarity and keyword matching; Embedding storage for AI agents and RAG pipelines

mogui-ADE-orchestrator

A workspace orchestration harness that runs a single AI agent as a master controller over multiple worker agents within the Orca ADE environment.

ProblemThe difficulty of manually driving and coordinating multiple AI agents running in separate terminal sessions, including issues with context loss, session termination, and lack of automated verification for worker output.

Use it forOrchestrating multiple concurrent AI coding agents in a shared workspace; Managing long-running agent tasks that require context succession and state persistence; Coordinating work across isolated Git worktrees using terminal-based agent sessions

SWT-Bench

SWT-Bench is a NeurIPS 2024 evaluation harness for benchmarking LLMs on repository-level test generation.

ProblemLacks a standardized, reproducible benchmark to evaluate LLMs on generating tests that correctly identify and validate real-world bug fixes in existing codebases.

Use it forEvaluating the effectiveness of LLMs in generating reproducing tests for real-world bugs; Benchmarking code agents on their ability to write tests that validate issue resolutions; Comparing different LLMs and agents on the SWT-Bench Lite and Verified leaderboards

Token-Efficient Data Reasoning Agents via Adaptive Structuring of Unstructured Data

This paper proposes 'agentic data cracking,' a method that adaptively structures unstructured data as a byproduct of LLM agent reasoning.

ProblemLLM agents currently consume prohibitive amounts of tokens (up to a million per question) by repeatedly opening large unstructured documents to recover scattered evidence, whereas structured data would allow for cheap da

Use it forReducing token costs for enterprise AI agents answering questions over large document collections; Optimizing reasoning over unstructured sources like web pages, contracts, and earnings calls; Building shared data infrastructure for agentic reasoning where knowledge accumulates over time

Agent Detective

Agent Detective is an evaluation framework for multi-agent systems that ingests OpenTelemetry traces to reconstruct execution graphs and identify the specific node where quality failures originate.

ProblemMulti-agent systems are difficult to debug because failures often propagate downstream, making it hard to identify the original source of an error or quality degradation.

Use it forDebugging multi-agent pipelines by identifying the specific agent step that caused a downstream failure; Gating CI/CD pipelines for LLM applications based on trace quality and contract adherence; Visualizing agent execution graphs to understand how errors propagate through parallel branches

Agent Nexus

Agent Nexus is a self-hosted, model-agnostic AI agent orchestration platform built with Go, PostgreSQL, and Next.js.

ProblemLack of a unified, self-hosted platform to orchestrate, observe, and manage AI agents across different LLM providers and tool ecosystems without vendor lock-in.

Use it forOrchestrating multi-step AI agent workflows with visual editing; Automating Jira ticket to Pull Request pipelines using headless coding agents; Integrating external tools via Model Context Protocol (MCP) servers

DriftWatch: Data Drift Detector for ML Production

DriftWatch is a lightweight Python library and CLI tool for detecting data drift in machine learning production environments.

ProblemML models often degrade silently when input data drifts from the training distribution, and existing monitoring solutions can be heavy or complex to integrate.

Use it forMonitoring feature distribution shifts in live ML models; Generating automated drift reports for incident investigation; Exposing drift metrics to Prometheus for dashboarding

Sandbox Orchestrator

A Python library for orchestrating ephemeral sandboxes to support large-scale reinforcement learning rollouts for coding agents.

ProblemTraining autonomous coding agents via RL requires running massive numbers of isolated, stateful sandboxes; current methods lack efficient mechanisms for instant provisioning, state migration during preemption, and cost o

Use it forSimulating large-scale RL training environments for autonomous coding agents; Prototyping sandbox lifecycle management and state checkpointing logic; Demonstrating cost-optimization strategies using spot instances

TreeTrace

TreeTrace is a local-first CLI tool that parses AI coding agent session transcripts to identify security risks, human corrections, and token usage.

ProblemAI coding agent sessions contain valuable steering and correction data that is lost when the session ends, and existing methods for evaluating agent behavior often rely on non-deterministic LLM judges or require uploadin

Use it forGenerating local regression evals from human corrections of AI coding agents; Auditing AI agent sessions for security slips like exposed secrets or risky commands; Analyzing token efficiency and rework costs in AI-assisted development workflows

padosoft/eval-harness

A Laravel-native evaluation framework for RAG and LLM pipelines that uses golden datasets, LLM-as-judge metrics, and statistical regression detection.

ProblemStandard evaluation tools lack statistical rigor for detecting real changes in LLM outputs and do not natively integrate with Laravel's ecosystem or provide per-row regression tracking.

Use it forRegression testing for RAG pipelines in Laravel applications; Evaluating agent tool-calling trajectories and step budgets; CI/CD gating for LLM-based features using statistical confidence

Cabeza: Configurable Agentic Search Harness

Cabeza is a Python library providing a configurable inference harness for long-horizon agentic search tasks.

ProblemDeveloping and testing long-horizon agentic search systems requires complex, custom implementations for context management, memory, and multi-agent coordination, which are often tightly coupled to specific LLM providers.

Use it forBuilding autonomous agents that perform multi-step web research; Evaluating different LLM providers and context management strategies for search tasks; Implementing long-horizon agents with persistent memory and summarization

llmff

llmff is a Rust-based command-line tool and library for defining LLM inference pipelines using YAML manifests.

ProblemLack of a standardized, inspectable, and reproducible tool for defining and debugging LLM inference pipelines, particularly for local/offline execution.

Use it forBuilding reproducible LLM workflows with JSON validation and repair stages; Running offline LLM pipelines against local Ollama instances; Debugging LLM inference with detailed JSONL trace logs

EMU: Efficient Episodic Memory Utilization for Cooperative MARL

This is the official PyTorch implementation of the EMU algorithm, a method for cooperative multi-agent reinforcement learning that utilizes episodic memory.

ProblemStandard cooperative MARL methods often struggle with sample efficiency and long-horizon credit assignment; EMU addresses this by efficiently utilizing episodic memory to guide learning.

Use it forTraining cooperative multi-agent policies in StarCraft II (e.g., 5m_vs_6m); Training multi-agent policies in Google Research Football (e.g., 3_vs_1_with_keeper); Researching efficient episodic memory mechanisms in deep multi-agent reinforcement learning

EPLB: Expert Parallelism Load Balancer

A Python library that implements load balancing algorithms for Mixture-of-Experts (MoE) models using expert parallelism.

ProblemUneven load distribution across GPUs in expert-parallel MoE deployments causes underutilization and performance bottlenecks.

Use it forBalancing expert load across GPUs in DeepSeek-V3 deployments; Optimizing MoE model inference performance in multi-node clusters; Implementing redundant expert strategies for large language models

Agentic LLM Vulnerability Scanner

A toolkit for performing red teaming and vulnerability scanning on agentic LLM systems.

ProblemLack of specialized tools for testing the security of autonomous LLM agents that interact with external tools.

Use it forSecurity auditing of LLM-based agents; Red teaming AI applications; Testing prompt injection vulnerabilities in agentic workflows

NVIDIA NeMo Guardrails

An open-source Python library for adding programmable guardrails to LLM-based conversational systems.

ProblemLLM-based applications lack built-in mechanisms to control output, stay on topic, or protect against adversarial attacks like prompt injection.

Use it forProtecting LLM chatbots against jailbreaks and prompt injection; Enforcing fact-checking in Retrieval Augmented Generation (RAG) applications; Steering domain-specific assistants to follow predefined conversational flows

skill-diffs

A large-scale dataset of agent skill (SKILL.md) commit histories and diffs scraped from public GitHub repositories across four platforms.

ProblemLack of high-quality, evolution-focused training data for agent skills, as existing datasets typically only capture static snapshots rather than the iterative refinement process.

Use it forFine-tuning LLMs to act as skill editors or curators; Training DPO or preference models on skill evolution; Benchmarking models on skill patching tasks

Predicting ICU Patient Clinical Deterioration Report

This is a research report and repository collection for predicting clinical deterioration in ICU patients using the MIMIC-III dataset.

ProblemTraditional severity scores like SAPS II may lack the flexibility to capture complex patient patterns, leading to suboptimal prediction performance.

Use it forAcademic research on ICU mortality prediction; Comparing flexible ML models against traditional severity scores; Exploratory data analysis of Electronic Health Records (EHR)

Agent Execution Engine

A self-hostable Python runtime for LLM agents that uses an explicit 12-state state machine to orchestrate tool calls, manage persistent memory, and provide observability.

ProblemHigh-level agent frameworks often lack the explicit control, observability, and reliability guarantees needed for production-grade autonomous agents.

Use it forBuilding reliable, observable AI agents that require complex tool orchestration; Implementing persistent agent memory with session and user-scoped storage; Integrating custom agent logic into existing systems via MCP or REST APIs

ABEX: Adaptive Strategy Generation for Boundary Value Exploration

ABEX is an agentic LLM framework that automates Boundary Value Exploration (BVE) by replacing hand-engineered mutation operators with adaptive, natural-language strategies.

ProblemExisting automated Boundary Value Exploration techniques rely on hand-engineered mutation operators specific to each input type or function, limiting their applicability to numeric inputs and making them ineffective for

Use it forAutomated black-box testing of software functions with mixed input types (numeric, string, array); Discovering fault-prone boundary conditions in legacy code without access to source code; Generating high-quality test suites that reveal stubborn mutants

Sustainability of Open-Source ML Robustness Assessment Tools

This paper presents an empirical study of the open-source machine learning robustness tooling ecosystem.

ProblemPractitioners lack visibility into the maintenance status and long-term sustainability of open-source ML robustness tools they rely on for evaluation and assurance.

Use it forAssessing the maintenance risk of ML robustness libraries before adoption; Identifying active vs. inactive tools for AI assurance compliance; Understanding the sustainability landscape of adversarial testing frameworks

DeLLMGuard: Defending Smart Contracts Against LLM-Based Vulnerability Scanning

This paper proposes DeLLMGuard, a smart contract deployment framework that defends against malicious LLM-based vulnerability scanning.

ProblemPublicly disclosed smart contract source code is increasingly being used by LLM agents to scan for vulnerabilities and exploits at scale, creating a new threat model for blockchain security.

Use it forProtecting smart contract source code from automated LLM vulnerability scanning; Maintaining public source code disclosure for trust while mitigating automated exploit discovery; Securing financial smart contracts against large-scale agent-based attacks

Recovering Software Architecture Intent from Historical Work Items using Generative AI

This paper presents a five-step LLM-based workflow that extracts C4 architecture diagrams from historical agile work items in Azure DevOps.

ProblemSoftware architecture design intent is often only partially captured in code, while valuable traces remain in scattered, unstructured agile work items that do not directly support architectural analysis.

Use it forReconstructing missing or outdated architecture documentation from agile tracking data; Identifying architectural drift by comparing generated baselines with implemented code; System comprehension for new team members joining legacy projects

On the Maintenance and Co-evolution of Agent Plugins: An Empirical Study of Claude Code Plugin Marketplaces

This paper presents an empirical study of 1,926 repositories hosting Claude Code plugin marketplaces, analyzing 8,351 plugins and 77,773 commits.

ProblemThe lack of empirical understanding regarding the structure, maintenance practices, and co-evolution dynamics of agent plugin repositories, which function differently from traditional source-code-based software packages.

Use it forUnderstanding the maintenance lifecycle of AI agent plugins; Benchmarking plugin development activity against traditional open-source software; Identifying co-evolution patterns between instruction files and implementation scripts

T-Code

T-Code is a Java-based coding agent CLI that integrates ReAct, Plan-and-Execute, and Multi-Agent execution modes.

ProblemLack of a unified, secure, and flexible local coding agent that supports multiple execution paradigms (ReAct, Planning, Multi-Agent) and integrates with existing MCP tools.

Use it forAutomating complex multi-step code refactoring tasks using Plan-and-Execute mode; Collaborative code development using Multi-Agent teams for planning, execution, and review; Interactive code exploration and debugging via terminal with LSP diagnostics

Crab Code

Crab Code is an open-source agentic coding CLI written in Rust that serves as an alternative to Claude Code.

ProblemDevelopers seeking a fast, open-source, and multi-provider alternative to proprietary agentic coding CLIs like Claude Code.

Use it forAutomating code refactoring and bug fixing in a terminal environment; Interacting with local codebases using various LLM providers; Running agentic coding workflows with a permission-based security model

Introducing Breeze TTS 2

This is a product announcement blog post for Breeze TTS 2, a new speech model designed for real-time interactive voice AI experiences.

ProblemStandard TTS models lack the diversity, controllability, and low latency required for real-time interactive applications like games and voice agents.

Use it forCreating distinct voices for game NPCs and characters; Building real-time voice agents and interactive companions; Producing interactive stories and dynamic media content

ragproof: RAG Evaluation Harness

ragproof is an open-source Python library and CLI tool for evaluating Retrieval-Augmented Generation (RAG) systems.

ProblemMost RAG failures are actually retrieval failures, but teams often lack the tools to measure retrieval quality or detect when it degrades, leading to undiagnosed performance drops.

Use it forBenchmarking RAG retrieval performance on standard datasets like BEIR/scifact; Detecting query drift and retrieval regressions in production RAG apps; Evaluating the faithfulness of LLM-generated answers against retrieved context

CapyMOA

CapyMOA is a Python library for efficient machine learning on data streams, providing tools for classification, regression, clustering, and drift detection.

ProblemExisting Python libraries for data stream mining often lack the performance of dedicated tools, while high-performance tools like MOA lack the flexibility and ecosystem integration of Python.

Use it forReal-time anomaly detection in sensor data; Online classification of streaming text or network traffic; Drift detection in financial time series

Gnomon

Gnomon is a Python library designed to add statistical rigor to LLM agent evaluations.

ProblemMost LLM evaluation tools report point estimates without uncertainty, leading teams to make shipping decisions based on differences that are within the margin of error.

Use it forDetermining if a new LLM agent version is statistically significantly better than the previous one; Calculating required sample sizes for A/B tests to detect specific effect sizes; Calibrating LLM judges by measuring agreement with human raters using Cohen's kappa

cinfer

A lightweight deep learning inference engine written in C.

ProblemStandard deep learning inference frameworks are often too resource-intensive or complex for lightweight, C-based, or embedded applications.

Use it forRunning deep learning models in resource-constrained environments; Integrating inference capabilities into C-based applications; Deploying models on embedded systems or edge devices

VeritasRAG

VeritasRAG is a multi-agent Retrieval-Augmented Generation system built with LangChain and LangGraph that uses a stateful reflection loop to verify answers.

ProblemTraditional RAG pipelines often produce hallucinated answers because they perform a single retrieval-generation pass without validating the output against the source context.

Use it forReducing hallucinations in technical documentation Q&A; Building self-correcting RAG pipelines for local LLMs; Interactive document querying with confidence scoring via Gradio

IaCBench: Benchmark for LLM-Generated Infrastructure-as-Code

This repository provides a benchmark of 150 Infrastructure-as-Code tasks (Kubernetes, Terraform, Dockerfile) and an evaluation harness called InfraAgent.

ProblemLack of a rigorous, multi-language benchmark to quantify how reliably LLMs generate secure and functional Infrastructure-as-Code, particularly regarding security misconfigurations.

Use it forEvaluating the reliability of LLMs for generating secure infrastructure code; Comparing security compliance rates between different LLM providers; Reproducing academic research on LLM performance in DevOps tasks