Data & Analytics

44 of 44 repositories

• microsoft/markitdown

  Python tool for converting files and office documents to Markdown.

  AI summary: A lightweight Python utility for converting various file formats into clean Markdown.

  172,163dataPythonMITautogenautogen-extensionlangchainmarkdown

• firecrawl/firecrawl

  The context API to search, scrape, and interact with the web at scale. 🔥

  AI summary: An API service that crawls websites and turns them into clean, LLM-ready markdown data.

  162,732dataTypeScriptAGPL-3.0aiai-agentsai-crawlerai-scraping

• immich-app/immich

  High performance self-hosted photo and video management solution.

  AI summary: High-performance self-hosted alternative to Google Photos for managing personal media.

  109,908dataTypeScriptAGPL-3.0backup-toolfluttergoogle-photosgoogle-photos-alternative

• opendatalab/MinerU

  Transforms complex documents like PDFs and Office docs into LLM-ready markdown/JSON for your Agentic workflows.

  AI summary: High-accuracy parsing engine that converts complex documents into LLM-ready Markdown and JSON.

  77,081dataPythonOtherai4sciencedocument-analysisdocxextract-data

• D4Vinci/Scrapling

  🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!

  AI summary: A fast, undetectable web scraping framework for Python.

  72,975dataPythonBSD-3-Clauseaiai-scrapingautomationcrawler

• OpenBB-finance/OpenBB

  Open Data Platform for analysts, quants and AI agents.

  AI summary: An open data platform providing comprehensive financial data for analysts, quants, and AI agents.

  71,496dataPythonOtheraicryptoderivativeseconomics

• virattt/ai-hedge-fund

  An AI Hedge Fund Team

  AI summary: An experimental framework for simulating algorithmic trading strategies using multi-agent LLM consensus.

  62,685dataPythonMITalgorithmic-tradingfinancemulti-agentllm

• sansan0/TrendRadar

  ⭐AI-driven public opinion & trend monitor with multi-platform aggregation, RSS, and smart alerts.🎯 告别信息过载，你的 AI 舆情监控助手与热点筛选工具！聚合多平台热点 + RSS 订阅，支持关键词精准筛选。AI 智能筛选新闻 + AI 翻译 + AI 分析简报直推手机，也支持接入 MCP 架构，赋能 AI 自然语言对话分析、情感洞察与趋势预测等。支持 Docker ，数据本地/云端自持。集成微信/飞书/钉钉/Telegram/邮件/ntfy/bark/slack 等渠道智能推送。

  AI summary: An automated intelligence gathering tool that monitors and summarizes emerging tech trends across platforms.

  61,197dataPythonGPL-3.0aibarkdata-analysisdocker

• ZhuLinsen/daily_stock_analysis

  LLM 驱动的多市场股票智能分析系统：多源行情、实时新闻、决策看板与自动推送，支持零成本定时运行。 LLM-powered multi-market stock analysis system with multi-source market data, real-time news, decision dashboard, automated notifications, and cost-free scheduled runs.

  AI summary: An automated system for scraping, analyzing, and visualizing daily A-share stock market data.

  60,283dataPythonMITa-stockai-agentaigcllm

• NanmiCoder/MediaCrawler

  小红书笔记 | 评论爬虫、抖音视频 | 评论爬虫、快手视频 | 评论爬虫、B 站视频 ｜ 评论爬虫、微博帖子 ｜ 评论爬虫、百度贴吧帖子 ｜ 百度贴吧评论回复爬虫 | 知乎问答文章｜评论爬虫

  AI summary: A comprehensive crawler for extracting posts and comments from major Chinese social media platforms.

  60,171dataPythonOthercrawlerscrapersocial-mediadata-extraction

• KeygraphHQ/shannon

  Shannon is an AI pentester for web applications and APIs. It analyzes your source code, identifies attack vectors, and executes real exploits to prove vulnerabilities before they reach production.

  AI summary: A high-performance, open-source knowledge graph engine optimized for fast querying and AI integration.

  46,488dataTypeScriptAGPL-3.0agentsai-penetration-testingai-securitycybersecurity

• google/langextract

  A Python library for extracting structured information from unstructured text using LLMs with precise source grounding and interactive visualization.

  AI summary: AI-powered tool for extracting structured data from unstructured text using large language models.

  37,990dataPythonApache-2.0geminigemini-aigemini-apigemini-flash

• Fincept-Corporation/FinceptTerminal

  FinceptTerminal is a modern finance application offering advanced market analytics, investment research, and economic data tools, designed for interactive exploration and data-driven decision-making in a user-friendly environment.

  AI summary: A native C++ financial intelligence platform combining institutional analytics with local AI agents.

  29,914dataC++Otherai-agentsalgorithmic-tradingbloomberg-terminalcpp

• topoteretes/cognee

  Cognee is the open-source AI memory platform for agents. Give your AI agents persistent long-term memory across sessions with a self-hosted knowledge graph engine.

  AI summary: An open-source AI memory platform that provides persistent long-term memory and knowledge graphs for agents.

  29,811dataPythonApache-2.0agent-memoryagent-skillsaiai-agents

• opendataloader-project/opendataloader-pdf

  PDF Parser for AI-ready data. Automate PDF accessibility. Open-source.

  AI summary: A high-accuracy PDF parser that extracts AI-ready Markdown, JSON, and HTML using deterministic and hybrid approaches.

  28,188dataJavaApache-2.0a11yaccessibilityaibounding-box

• OtterMind/Chat2DB

  🔥🔥🔥 AI-driven database tool and SQL client, The hottest GUI client, supporting MySQL, Oracle, PostgreSQL, DB2, SQL Server, DB2, SQLite, H2, ClickHouse, and more.

  AI summary: An AI-driven database tool and SQL client that supports natural language queries.

  27,781dataJavaOtheraibichatgptclickhouse

• nautechsystems/nautilus_trader

  Production-grade Rust-native trading engine with deterministic event-driven architecture

  AI summary: A highly deterministic, Rust-native algorithmic trading engine for multiple asset classes.

  25,339dataRustLGPL-3.0algorithmic-trading-engineartificial-intelligencecrypto-tradingequity-trading

• hummingbot/hummingbot

  Open source software that helps you create and deploy high-frequency crypto trading bots

  AI summary: An open-source framework for building and deploying high-frequency cryptocurrency trading bots across CeFi and DeFi.

  19,363dataPythonApache-2.0ai-agentsalgotradingarbitragebacktesting

• hasaneyldrm/exercises-dataset

  1,324-exercise fitness dataset — animation GIFs, 180×180 thumbnails, muscle-group & equipment data, and step-by-step instructions in 6 languages. The exercise data layer behind the LogPress app.

  AI summary: A comprehensive JSON dataset of over 1,300 physical exercises.

  19,089dataHTMLOtherdatasetexercise-databaseexercisesfitness

• alibaba/zvec

  A lightweight, lightning-fast, in-process vector database

  AI summary: High-performance C++ vector database optimized for local embedded similarity search.

  15,398dataC++Apache-2.0agent-skillsdbembeddedfaiss

• xbtlin/ai-berkshire

  AI 时代的伯克希尔：基于 Claude Code / Codex 的价值投资研究框架。巴菲特·芒格·段永平·李录四大师方法论 + 多Agent并行研究。| AI-era Berkshire: a value investing research framework built for Claude Code / Codex. 4 masters' methodologies + multi-agent adversarial analysis.

  AI summary: An AI-driven value investing research framework combining multiple expert methodologies.

  15,087dataPythonMITaiai-agentanthropicberkshire-hathaway

• eigent-ai/eigent

  Eigent: The Open Source Cowork Desktop - Local and Free Alternative to Claude Cowork and Codex

  AI summary: An intelligent data extraction and transformation pipeline for unstructured enterprise documents.

  14,783dataTypeScriptApache-2.0agent-frameworkagent-skillsagentic-aiagentic-workflow

• RyanCodrai/turbovec

  A vector index built on TurboQuant, written in Rust with Python bindings

  AI summary: Rust vector index with Python bindings based on TurboQuant for extreme RAM compression.

  14,655dataRustMITannavx512embeddingembeddings

• wechat-article/wechat-article-exporter

  一款在线的 微信公众号文章批量下载 工具，支持导出阅读量与评论数据，无需搭建任何环境，可通过 在线网站 使用，支持 docker 私有化部署和 Cloudflare 部署。 支持下载各种文件格式，其中 HTML 格式可100%还原文章排版与样式。

  AI summary: A comprehensive tool to bulk download WeChat public articles and export their data.

  12,670dataTypeScriptMITdownloadwechatwechat-articlewechat-download

• run-llama/liteparse

  A fast, helpful, and open-source document parser

  AI summary: A fast, lightweight, and open-source Rust tool for high-quality spatial PDF parsing.

  11,931dataRustApache-2.0document-ocrdocument-processingocrocr-recognition

• calesthio/Crucix

  Your personal intelligence agent. Watches the world from multiple data sources and pings you when something changes.

  AI summary: A high-performance, distributed key-value store optimized for read-heavy workloads.

  11,176dataJavaScriptAGPL-3.0aiintelligenceosintdatabase

• BigBodyCobain/Shadowbroker

  Open-source intelligence for the global theater. Track everything from the corporate/private jets of the wealthy, and spy satellites, to seismic events in one unified interface. Hook an AI agent up to have it parse through data and find previously unseen correlations. The knowledge is available to all but rarely aggregated in the open, until now.

  AI summary: A decentralized, real-time geospatial intelligence platform for tracking global assets.

  10,595dataPythonAGPL-3.0ads-badsbaircraftaircraft-tracking

• apache/cassandra

  Open source transactional distributed database. Linear scalability and proven fault-tolerance on commodity hardware or cloud infrastructure without compromising performance.

  AI summary: Highly scalable, partitioned row store database designed for zero single points of failure.

  10,058dataJavaApache-2.0cassandradatabasejavanosql

• TencentCloud/TencentDB-Agent-Memory

  TencentDB Agent Memory is a team-level memory hub for AI Agents — turning conversations, docs, and code into four reusable memory assets (Chat Memory, Skill, LLM-Wiki, Code-Graph) that are governed, shared, and equipped across agents and frameworks.

  AI summary: A team-level memory hub governing chat, skills, wikis, and code graphs for AI agents.

  9,340dataTypeScriptOtheragentai-agentembeddingllm

• xberg-io/xberg

  A polyglot document intelligence framework with a Rust core. Extract text, metadata, images, and structured data from 101 formats (115 file extensions) plus code intelligence for 371 code languages. 15 language bindings — Rust, Python, Ruby, Java, Go, PHP, Elixir, C#, TypeScript — plus CLI, REST API, and MCP server.

  AI summary: High-performance polyglot document extraction framework built in Rust.

  8,910dataRustMITbuncsharpdocument-intelligenceelixir

• xberg-io/xberg

  A polyglot document intelligence framework with a Rust core. Extract text, metadata, images, and structured information from PDFs, Office documents, images, and 98+ formats. Available for Rust, Python, Ruby, Java, Go, PHP, Elixir, C#, R, C, TypeScript (Node/Bun/Wasm/Deno)- or use via CLI, REST API, or MCP server.

  AI summary: A unified document intelligence engine for extracting structured data from 98 file formats.

  8,712dataRustMITbuncsharpdocument-intelligenceelixir

• GoogleCloudPlatform/knowledge-catalog

  Google Cloud Knowledge Catalog Tools and Samples

  AI summary: An AI-powered data catalog and metadata management platform for Google Cloud.

  8,346dataTypeScriptApache-2.0data-catalogmetadata-managementknowledge-graphgoogle-cloud

• medialab/xan

  The CSV magician

  AI summary: A blazingly fast CSV processing CLI written in Rust.

  4,450dataRustUnlicenseclicsvrusttsv

• malisper/pgrust

  Postgres rewritten in Rust, now faster than Postgres and Clickhouse

  AI summary: A high-performance Rust rewrite of Postgres designed for deep, AI-assisted database customization.

  4,026dataRustAGPL-3.0ai-assisted-developmentdatabasepostgrespostgresql

• rahulnyk/knowledge_graph

  Convert any text to a graph of knowledge. This can be used for Graph Augmented Generation or Knowledge Graph based QnA

  AI summary: A tool to convert unstructured text into knowledge graphs for enhanced QnA and data generation.

  3,621dataJupyter NotebookMITknowledge-graphnlpragdata-extraction

• russellromney/honker

  SQLite extension + bindings for Postgres NOTIFY/LISTEN semantics with durable queues, streams, pub/sub, and scheduler

  AI summary: A SQLite extension bringing durable pub/sub, task queues, and Postgres-style NOTIFY/LISTEN semantics.

  2,974dataPythonOthersqlitedatabase-extensionpubsubtask-queue

• microsoft/pg_durable

  PostgreSQL in-database durable execution

  AI summary: A PostgreSQL extension that provides durable, transactionally-safe queues and background jobs.

  2,735dataRustOtherai-pipelinesai-workflowsdurable-executiondurable-functions

• huangserva/3DCellForge

  AI-powered interactive 3D model generation, inspection, and presentation studio.

  AI summary: An open-source toolkit for generating and analyzing 3D cellular models from microscopy data.

  2,588dataJavaScriptMITbioinformatics3d-modelingmicroscopyimage-processing

• microsoft/Ontology-Playground

  Free, open-source web app for learning about ontologies and Microsoft Fabric IQ. Explore a catalogue of pre-built ontologies, design your own visually, export as RDF/XML, and share interactive diagrams. Zero backend, fully static.

  AI summary: Free, open-source static web app for learning about ontologies and Microsoft Fabric IQ.

  2,356dataTypeScriptMITontologysemantic-webdata-modelingmicrosoft-fabric

• tnm/zclaw

  Your personal AI assistant at all-in 888KiB (~35KB in app code). Running on an ESP32. GPIO, cron, custom tools, memory, and more.

  AI summary: ZClaw is a highly concurrent, distributed web scraper built for massive data extraction.

  2,203dataCMITweb-scrapingdistributed-systemsconcurrencydata-extraction