microsoft/onnxruntimePublic

What is onnxruntime?

A cross-platform, high-performance machine learning inferencing and training accelerator.

What does onnxruntime do?

ONNX Runtime is a robust engine designed to execute machine learning models across diverse hardware platforms with maximum efficiency. It accelerates inference by natively supporting models trained in frameworks like PyTorch, TensorFlow, and scikit-learn via the Open Neural Network Exchange (ONNX) format. The runtime optimizes execution through advanced graph transformations and dynamically leverages hardware-specific execution providers, such as TensorRT for NVIDIA GPUs or CoreML for Apple Silicon. Beyond inferencing, it also provides tools to significantly accelerate model training times on multi-node GPU clusters.

Who is onnxruntime for?

Machine learning engineers, AI researchers, and production deployment teams looking to optimize and standardize model inference across varied hardware ecosystems. It provides a reliable pathway to deploy deep learning models seamlessly across operating systems.

How do I get started with onnxruntime?

Install via package managers (e.g., pip install onnxruntime) or refer to the official documentation for language-specific bindings.

How popular is onnxruntime on GitHub?

microsoft/onnxruntime has 22,151 stars and 4,358 forks on GitHub, and gained 70 stars in the last 7 days.

What license does onnxruntime use?

microsoft/onnxruntime is released under the MIT license.

• Oct 12, 2026New release v1.31.0 (previously v1.30.0).

• Widely adopted

  22,151 stars

• Battle-tested

  7 years of history

• Very active

  3,384 commits in 52 weeks

• Community-driven

  ~1,027 contributors

• Well documented

  High community health score

• Permissive license

  MIT

ONNX Runtime is a robust engine designed to execute machine learning models across diverse hardware platforms with maximum efficiency. It accelerates inference by natively supporting models trained in frameworks like PyTorch, TensorFlow, and scikit-learn via the Open Neural Network Exchange (ONNX) format. The runtime optimizes execution through advanced graph transformations and dynamically leverages hardware-specific execution providers, such as TensorRT for NVIDIA GPUs or CoreML for Apple Silicon. Beyond inferencing, it also provides tools to significantly accelerate model training times on multi-node GPU clusters.

Machine learning engineers, AI researchers, and production deployment teams looking to optimize and standardize model inference across varied hardware ecosystems. It provides a reliable pathway to deploy deep learning models seamlessly across operating systems.

• Hardware Acceleration: Utilize specialized execution providers (like CUDA, TensorRT, or DirectML) to maximize performance on specific hardware.
• Framework Agnostic: Seamlessly run models trained in PyTorch, TensorFlow/Keras, and classical ML libraries without framework lock-in.
• Graph Optimization: Automatically apply graph transforms and optimizations to reduce execution latency and memory footprint during inference.
• Cross-Platform Execution: Deploy the exact same optimized model across Windows, Linux, macOS, Android, and iOS environments.
• Training Acceleration: Speed up transformer model training on multi-node NVIDIA GPU setups with minimal changes to existing PyTorch scripts.

Getting started: Install via package managers (e.g., pip install onnxruntime) or refer to the official documentation for language-specific bindings.

1. ONNX Runtime v1.31.0v1.31.0Oct 9, 202619.7K downloads

   ONNX Runtime 1.31.0 stabilizes model-package and EPContext data APIs, improves CPU model loading and quantized MoE inference, expands device-based execution-provider selection, and strengthens model-loading and runtime reliability. These notes cover changes since ONNX Runtime 1.30.0. CUDA and WebGPU kernel updates are summarized here; detailed provider notes are covered by their separate plugin EP releases. ## Highlights - Promoted the model-package API and EPContext data callbacks to stable C and C++ APIs, with EPContext callback support added across language bindings ([#33166](https://github.com/microsoft/onnxruntime/pull/33166), [#32265](https://github.com/microsoft/onnxruntime/pull/32265)). - Reduced CPU session-initialization overhead by parallelizing eligible weight prepacking, and accelerated block-wise INT4/INT8 QMoE experts with MLAS QNBit kernels and grouped expert dispatch ([#31691](https://github.com/microsoft/onnxruntime/pull/31691), [#32644](https://github.com/microsoft/onnxruntime/pull/32644), [#32668](https://github.com/microsoft/onnxruntime/pull/32668)). - Added x86 FP16 LayerNorm/RMSNorm acceleration, Arm KleidiAI SVE2.1 FP16 GEMM support, and additional R

2. ONNX Runtime CUDA Plugin EP 0.3.0plugin-ep-cuda/v0.3.0Oct 9, 2026170 downloads

   This release brings sparse attention and packed continuous-batching support, INT2 and mixed-width QMoE execution, decode-kernel tuning, and reliability fixes to the ONNX Runtime CUDA Plugin EP. Highlights include new QSA/CSA attention operators, expanded paged speculative decoding, opt-in MatMul auto-tuning, and improved MXFP4 support on Blackwell consumer GPUs. ## Compatibility The minimum ONNX Runtime core version remains **1.24.4**, but meeting that minimum alone does not guarantee compatibility for contributed operators. Core and the plugin must use the same contributed-operator schemas; building both from the **same ONNX Runtime revision** is recommended. Schema mismatches can cause incorrect execution or crashes and may not be detected when registering the plugin ([#32855](https://github.com/microsoft/onnxruntime/pull/32855)). The matched ONNX Runtime version is 1.31.0 for this release. ## Highlights ### Plugin Runtime and CUDA Graphs - Added checked `GroupQueryAttention` workspace recipes for KV-cache preparation and XQA, FlashAttention, memory-efficient, and unfused routes, plus conservative workspace estimation for resource-constrained graph partiti

3. ONNX Runtime v1.28.3v1.28.3Oct 2, 20267.4K downloads

   This is a patch release on top of [v1.28.2](https://github.com/microsoft/onnxruntime/releases/tag/v1.28.2), containing memory-safety and model-validation fixes, a plugin execution fix, and packaging infrastructure updates. ## Highlights ### Bug Fixes - Fixed an out-of-bounds read in `EmbedLayerNormalization` shape inference when beta is a scalar ([#32610](https://github.com/microsoft/onnxruntime/pull/32610)). - Fixed a heap buffer overflow when reusing a packed sub-byte buffer for a full-byte tensor ([#32611](https://github.com/microsoft/onnxruntime/pull/32611)). - Fixed an out-of-bounds read when a Loop or Scan body input is also an initializer ([#32609](https://github.com/microsoft/onnxruntime/pull/32609)). - Hardened shape inference for applicable contrib operators, including GroupQueryAttention, SparseAttention, and CausalConvWithState ([#32607](https://github.com/microsoft/onnxruntime/pull/32607)). - Fixed out-of-bounds reads in ConvTransposeWithDynamicPads and `InferenceContextImpl::getInputData` ([#32679](https://github.com/microsoft/onnxruntime/pull/32679), [#32681](https://github.com/microsoft/onnxruntime/pull/32681)). - Rejected invalid graph structures: inp

4. ONNX Runtime CUDA Plugin EP 0.2.0plugin-ep-cuda/v0.2.0Oct 8, 202639 downloads

   This release brings expanded quantized inference and attention support, kernel performance improvements, and reliability fixes to the ONNX Runtime CUDA Plugin EP. Highlights include 2-bit `MatMulNBits`, INT4 paged KV caches, speculative decoding improvements, compact recurrent-state updates, and safer CUDA Graph execution across sessions. Please refer to the [CUDA Plugin EP Quick Start](https://github.com/microsoft/onnxruntime/blob/plugin-ep-cuda/rel-0.2/docs/cuda_plugin_ep/QUICK_START.md) for usage and installation guidance. The matched ONNX Runtime version is 1.30.0. ## Highlights ### Plugin Runtime and CUDA Graphs - Gave each plugin EP session its own device arena, preventing cross-session reuse of CUDA Graph buffers and honoring per-session arena options ([#32807](https://github.com/microsoft/onnxruntime/pull/32807)). - Added workspace-memory accounting and reporting for resource-constrained graph partitioning, including packed attention workspace recipes and estimates, and optional-input-aware shape handling ([#31962](https://github.com/microsoft/onnxruntime/pull/31962), [#32283](https://github.com/microsoft/onnxruntime/pull/32283), [#32312](https://github.com/m

5. ONNX Runtime v1.30.0v1.30.0Sep 10, 2026217.9K downloads

   ONNX Runtime 1.30.0 expands generative AI inference, improves CPU and GPU performance, adds Go bindings, and strengthens runtime reliability. These notes cover changes since ONNX Runtime 1.29.1. ## Highlights - Expanded CUDA inference support with variable-length causal convolution for continuous batching, speculative decoding in paged XQA, and INT4 paged KV caches with per-channel scales ([#32168](https://github.com/microsoft/onnxruntime/pull/32168), [#32340](https://github.com/microsoft/onnxruntime/pull/32340), [#32515](https://github.com/microsoft/onnxruntime/pull/32515)). - Improved WebGPU PagedAttention, added GPT-OSS support and INT8 KV-cache block quantization, and extended convolution optimizations ([#31727](https://github.com/microsoft/onnxruntime/pull/31727), [#32277](https://github.com/microsoft/onnxruntime/pull/32277), [#32284](https://github.com/microsoft/onnxruntime/pull/32284), [#32420](https://github.com/microsoft/onnxruntime/pull/32420)). - Added fused CPU LinearAttention kernels for AVX-512, Arm64 NEON, and SVE, plus AVX2 LayerNorm/RMSNorm acceleration ([#31674](https://github.com/microsoft/onnxruntime/pull/31674), [#31973](https://github.com/microsoft/onn

• tensorflow/tensorflow

  An Open Source Machine Learning Framework for Everyone

  201.3K stars · C++

• f/prompts.chat

  f.k.a. Awesome ChatGPT Prompts. Share, discover, and collect prompts from the community. Free and open source — self-host for your organization with complete privacy.

  174.5K stars · HTML

• huggingface/transformers

  🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.

  168.2K stars · Python

• Comfy-Org/ComfyUI

  The most powerful and modular diffusion model GUI, api and backend with a graph/nodes interface. The fastest local inference engine in the world.

  142.9K stars · Python

• ggml-org/llama.cpp

  LLM inference in C/C++

  132.3K stars · C++

• rasbt/LLMs-from-scratch

  Implement a ChatGPT-like LLM in PyTorch from scratch, step by step

  107.3K stars · Jupyter Notebook