EvalAITools Check a repo

How We Review AI Tools

AI tools are difficult to compare with a single score because the right product depends on the job. This page explains what each part of EvalAITools is based on: the articles, and the live data on the dashboard and tool pages. Last reviewed: 17 October 2026.

In short: the dashboard measures public signals (GitHub activity, downloads, security practices, prices, status pages). It does not test whether a tool works well, and we never present a measurement as a hands-on review.

What kinds of articles we publish

  • Explainers and news. These summarize official announcements, documentation, pricing pages and independent reports. They are based on published sources, and we say so in the article. We do not run our own tests for them.
  • Comparisons. These compare features, plans, limits and prices from official pages and independent write-ups. Unless an article states otherwise, it is not based on our own hands-on benchmark.
  • Test scenarios. Some articles include a procedure readers can run themselves. We label these clearly as scenarios and not as results.

What we look at in articles

  • Core capabilities and intended use
  • Pricing, limits and plan differences at the date we checked
  • Availability, regions and eligibility
  • Documented limitations, risks and privacy settings
  • Independent reports and evaluations, where they exist

Evidence and attribution

We separate vendor claims from independent findings. When a company reports a benchmark, we say it is company-reported. When we rely on another writer’s experience, we name the source, summarize it in our own words and say that we have not repeated it. We never present another person’s test as our own.

If we test something ourselves

When we do run a hands-on test, we will describe the product version, plan, date and method, and share the limits of what we found.

Keeping articles current

AI products change quickly. We update articles when major features, prices or availability change, and we note the date we checked key facts. Readers should still confirm details on the provider’s own page before making a business-critical decision.

Live data: open-source AI tools

The dashboard tracks more than 100 open-source AI projects. Everything below is collected automatically from public sources, each number is shown with its source and the time it was read, and nothing in it is a hands-on test.

Maintenance status

Each repository is read from the public GitHub API: commits on the default branch in the last 91 days and in the same 91 days a year earlier (all commits, including merge commits, so projects that merge pull requests with merge commits show higher counts than a count that leaves them out, as our dated articles do), stars, the date of the last commit, licence and the archived flag. The README is scanned once a day for notices such as “maintenance mode” or “no longer maintained”. Each tool gets the first label that applies:

  • Archived: the repository is archived on GitHub and read-only, or its README says the project has been archived and it has had fewer than 30 commits in 91 days.
  • Maintenance mode: the README says the project is no longer actively developed.
  • No recent commits: no commits in the last 91 days.
  • Slowing down: fewer than half of the commits of the same period a year earlier.
  • Active: none of the above.

A few projects do most of their work on a branch other than the default; where we know that, we count that branch and say so on the tool’s page.

How the health score works

The health score adds five parts, 100 points in all:

  • Activity, up to 35: commits in the last 91 days on a log scale, full marks at 600.
  • Trend, up to 20: this window’s commits as a share of the same window a year earlier, full marks at 100% or more.
  • Recency, up to 15: 15 if the last commit is within 7 days, 12 within 30 days, 7 within 91 days, 3 within 182 days.
  • Releases, up to 10: 10 if the latest GitHub release is under 91 days old, 7 under 182 days, 4 under a year. Projects that publish no GitHub releases get a neutral 5.
  • Community, up to 20: pull requests merged in the last 91 days on a log scale, full marks at 300.

An archived project scores 0, and a README maintenance notice caps the score at 25. Scores of 70 and above are shown as strong, 40 to 69 as moderate and below 40 as weak. Every tool page shows the points for each part. The score measures maintenance signals only: a mature, stable project can be healthy with few commits, and a busy project can still be risky.

Downloads and installs

  • Package downloads: last month’s downloads of the tool’s own package on PyPI, npm or crates.io, read once a day through ecosyste.ms. We only count the package the project itself publishes under its name; client libraries, plugins and third-party packages that point to the same repository are left out. Where a project publishes no such package, the dashboard shows a dash.
  • Docker Hub pulls: the all-time pull count of the project’s official image.
  • Homebrew installs: installs in the last 30 days from Homebrew’s public analytics, for tools that have a formula.

All of these include automated installs from CI pipelines, containers and mirrors, so they show how widely software is pulled, not how many people use it. Numbers from different registries are not comparable with each other. One reading is stored per day so trends and growth can be shown once enough days exist.

Security practices and adoption

  • OpenSSF Scorecard: an automated review, run by the Open Source Security Foundation, of how a repository is run: code review, branch protection, pinned dependencies, known vulnerabilities, security policy, signed releases and other checks, each scored 0 to 10. We show the published result and link to the full report. Not every repository is scanned; where no result exists we show nothing rather than estimate. Scorecard does not find bugs, and a low score does not mean the software is unsafe.
  • Dependents: the number of packages that depend on the tool’s package, from Google’s deps.dev, split into direct and indirect.
  • Release downloads: how many times the files attached to the latest GitHub release were downloaded. This matters most for desktop apps and binaries that ship installers as release files.

Alternatives, abandoned tools and comparisons

Alternatives pages group tools by the job they do. Within a group, tools are ranked by status first (active, then slowing down, then stopped), then by health score, then by commits in the last 91 days. The abandoned tools list shows every tracked project that is archived, in maintenance mode, without recent commits or slowing down. Comparison pages put two tools’ live numbers side by side. None of these rankings say which tool is better for a particular use; they show which projects are being looked after.

Live data: AI services

  • AI subscriptions. Prices are copied by hand from each provider’s own pricing or help page and re-checked weekly, and every change is logged with its date. If a page does not state a price clearly, we leave it out rather than guess.
  • Service status. Where a provider has a public status page we can read, we check it every 10 minutes. A service shows Up when its status page reports it operational; Degraded, Outage or Maintenance when the status page reports that; Incident when a provider that publishes only an incident feed has posted one in the last 3 hours; No reports when such a feed lists nothing recent; and Unknown when our last attempt to read the page failed. Incident counts use what the provider itself posted, and the same incident is counted once even when it is updated. Providers post at different levels of detail, so counts are a rough signal, not a reliability ranking.
  • Model API prices. The price per million tokens for hundreds of models comes from OpenRouter’s public model list, read twice a day. It is what OpenRouter charges, which can differ from a provider’s direct price.

Visitor ratings

Star ratings come from visitors, not from us, and are kept separate from the data. An average appears only after three ratings, and each visitor can rate a tool or service once every 30 days.

Corrections

If a number looks wrong, for example a package that is not the project’s own or a status that does not match the repository, please tell us through the contact page. We check the source, fix the mapping or the data, and the page updates on the next refresh.