{"slug":"lm-evaluation-harness-pt","name":"lm-evaluation-harness-pt","summary":"Open-source fork of EleutherAI's LM Evaluation Harness for evaluating Portuguese LLMs and chatbots. It is the evaluation suite for the Open Portuguese LLM Leaderboard and runs few-shot Portuguese tasks through a local lm_eval CLI, including direct-response scoring, chat-template ","publisher":"eduagarcia","homepage":"https://github.com/eduagarcia/lm-evaluation-harness-pt","kind":"harness","hasApi":false,"hasMcp":false,"costRange":"unknown","costNotes":"Pricing is not stated. The page only shows a public git clone and pip install, including optional extras for vLLM and API clients. GPU, cluster, and third-party model API costs are unknown.","functions":["evaluate","extract","memory","rank","generate","reason","test"],"slices":[{"functionId":"evaluate","label":"Portuguese LLM task suite","note":"Runs few-shot Portuguese evaluations such as ASSIN 2 RTE/STS, BlueX, ENEM, FaQuAD-NLI, HateBR, Portuguese hate speech, TweetSentBR, and OAB exams, reporting metrics including F1-macro, Pearson, and accuracy.","cost":"Unknown"},{"functionId":"evaluate","label":"Multi-backend model evaluation","note":"Evaluates Hugging Face models, vLLM batch inference, and closed models through LiteLLM APIs, with chat-template detection, direct text responses, reasoning extraction, and GPU batch-size controls.","cost":"Unknown"},{"functionId":"extract","label":"extract","note":"Inferred from executor","cost":"Unknown"},{"functionId":"memory","label":"memory","note":"Inferred from executor","cost":"Unknown"},{"functionId":"rank","label":"rank","note":"Inferred from executor","cost":"Unknown"},{"functionId":"generate","label":"generate","note":"Inferred from executor","cost":"Unknown"},{"functionId":"reason","label":"reason","note":"Inferred from executor","cost":"Unknown"},{"functionId":"test","label":"test","note":"Inferred from executor","cost":"Unknown"}],"interfaces":[],"composabilityRationale":"Composable as a local evaluation harness via the lm_eval CLI and Python package, with Hugging Face, vLLM, and LiteLLM model backends. The page says LiteLLM can call external model APIs; it does not state that this project exposes its own API or MCP server. Leaderboard submission is described only as a separate Submit page, with no interface contract given.","signals":[],"functionProbabilities":[{"functionId":"evaluate","value":0.8,"source":"https://github.com/eduagarcia/lm-evaluation-harness-pt","method":"lexical","observedAt":"2026-10-19T01:26:53.433Z"},{"functionId":"extract","value":0.8,"source":"https://github.com/eduagarcia/lm-evaluation-harness-pt","method":"lexical","observedAt":"2026-10-19T01:26:53.433Z"},{"functionId":"memory","value":0.8,"source":"https://github.com/eduagarcia/lm-evaluation-harness-pt","method":"lexical","observedAt":"2026-10-19T01:26:53.433Z"},{"functionId":"rank","value":0.8,"source":"https://github.com/eduagarcia/lm-evaluation-harness-pt","method":"lexical","observedAt":"2026-10-19T01:26:53.433Z"},{"functionId":"generate","value":0.72,"source":"https://github.com/eduagarcia/lm-evaluation-harness-pt","method":"lexical","observedAt":"2026-10-19T01:26:53.433Z"},{"functionId":"reason","value":0.72,"source":"https://github.com/eduagarcia/lm-evaluation-harness-pt","method":"lexical","observedAt":"2026-10-19T01:26:53.433Z"},{"functionId":"test","value":0.72,"source":"https://github.com/eduagarcia/lm-evaluation-harness-pt","method":"lexical","observedAt":"2026-10-19T01:26:53.433Z"}],"claimVerification":[{"key":"mcp","status":"unverified","asserted":"false","evidenceSource":"https://github.com/eduagarcia/lm-evaluation-harness-pt","method":"fallback"},{"key":"api","status":"unverified","asserted":"false","evidenceSource":"https://github.com/eduagarcia/lm-evaluation-harness-pt","method":"fallback"},{"key":"cost","status":"unverified","asserted":"unknown","evidenceSource":"https://github.com/eduagarcia/lm-evaluation-harness-pt","method":"fallback"},{"key":"license","status":"unverified","asserted":"","evidenceSource":"https://github.com/eduagarcia/lm-evaluation-harness-pt","method":"fallback"}],"tweets":[],"notes":[],"origin":"intake","createdAt":"2026-10-19T01:26:54.232Z","updatedAt":"2026-10-19T01:26:54.232Z","sources":[{"kind":"web","label":"executor","url":"https://github.com/eduagarcia/lm-evaluation-harness-pt","excerpt":"BranchesTags\n\n## Latest commit\n\n## History\n\n3,279 Commits\n\n## Folders and files\n\n## Repository files navigation\n\n# Language Model Evaluation Harness for Portuguese LLMs\n\n**A framework for evaluating Large Language Models (LLMs) in Portugues","at":"2026-10-19T01:26:54.232Z"}],"recordId":"cmp_lm-evaluation-harness-pt","claims":{"name":"lm-evaluation-harness-pt","summary":"Open-source fork of EleutherAI's LM Evaluation Harness for evaluating Portuguese LLMs and chatbots. It is the evaluation suite for the Open Portuguese LLM Leaderboard and runs few-shot Portuguese tasks through a local lm_eval CLI, including direct-response scoring, chat-template ","kind":"harness","publisher":"eduagarcia","homepage":"https://github.com/eduagarcia/lm-evaluation-harness-pt","costRange":"unknown","costNotes":"Pricing is not stated. The page only shows a public git clone and pip install, including optional extras for vLLM and API clients. GPU, cluster, and third-party model API costs are unknown.","functions":["evaluate","extract","memory","rank","generate","reason","test"],"slices":[{"functionId":"evaluate","label":"Portuguese LLM task suite","note":"Runs few-shot Portuguese evaluations such as ASSIN 2 RTE/STS, BlueX, ENEM, FaQuAD-NLI, HateBR, Portuguese hate speech, TweetSentBR, and OAB exams, reporting metrics including F1-macro, Pearson, and accuracy.","cost":"Unknown"},{"functionId":"evaluate","label":"Multi-backend model evaluation","note":"Evaluates Hugging Face models, vLLM batch inference, and closed models through LiteLLM APIs, with chat-template detection, direct text responses, reasoning extraction, and GPU batch-size controls.","cost":"Unknown"},{"functionId":"extract","label":"extract","note":"Inferred from executor","cost":"Unknown"},{"functionId":"memory","label":"memory","note":"Inferred from executor","cost":"Unknown"},{"functionId":"rank","label":"rank","note":"Inferred from executor","cost":"Unknown"},{"functionId":"generate","label":"generate","note":"Inferred from executor","cost":"Unknown"},{"functionId":"reason","label":"reason","note":"Inferred from executor","cost":"Unknown"},{"functionId":"test","label":"test","note":"Inferred from executor","cost":"Unknown"}],"interfaces":[],"composabilityRationale":"Composable as a local evaluation harness via the lm_eval CLI and Python package, with Hugging Face, vLLM, and LiteLLM model backends. The page says LiteLLM can call external model APIs; it does not state that this project exposes its own API or MCP server. Leaderboard submission is described only as a separate Submit page, with no interface contract given."},"relations":[{"from":"lm-evaluation-harness-pt","to":"best-stateful-sandboxes-for-code-execution-in-2026","kind":"fills-same-slot-as","note":"Shared functions evaluate, memory, rank, test. Can fill the same function slot."},{"from":"lm-evaluation-harness-pt","to":"fix-cold-start-latency-in-ai-sandboxes-techniques","kind":"fills-same-slot-as","note":"Shared functions evaluate, extract, reason, test. Can fill the same function slot."},{"from":"lm-evaluation-harness-pt","to":"harness-arena","kind":"fills-same-slot-as","note":"Shared functions evaluate, generate, rank, test. Can fill the same function slot."},{"from":"lm-evaluation-harness-pt","to":"terminal-bench-2-1-benchmark-leaderboard","kind":"fills-same-slot-as","note":"Shared functions evaluate, generate, reason, test. Can fill the same function slot."},{"from":"lm-evaluation-harness-pt","to":"chatgpt-scheduled-tasks","kind":"fills-same-slot-as","note":"Shared functions evaluate, reason, test. Can fill the same function slot."},{"from":"lm-evaluation-harness-pt","to":"ai-agent-sandbox-comparison-2026-seven-platforms","kind":"fills-same-slot-as","note":"Shared functions evaluate, memory, test. Can fill the same function slot."},{"from":"lm-evaluation-harness-pt","to":"airbnb-price-monitor","kind":"fills-same-slot-as","note":"Shared functions evaluate, extract, memory. Can fill the same function slot."},{"from":"lm-evaluation-harness-pt","to":"daytona-alternatives-4-ai-agent-sandbox-platforms-compared-2026","kind":"fills-same-slot-as","note":"Shared functions generate, memory, rank. Can fill the same function slot."}],"explain":{"score":3,"formula":"100 × sum(weight × value) over observed factors only. Unobserved factors are omitted, not zeroed.","factors":[{"id":"github_health","label":"GitHub health","weight":0.18,"why":"insufficient evidence","sources":[],"evidence":"insufficient"},{"id":"agent_use","label":"Agent use","weight":0.16,"why":"insufficient evidence","sources":[],"evidence":"insufficient"},{"id":"human_notes","label":"Human notes","weight":0.14,"why":"insufficient evidence","sources":[],"evidence":"insufficient"},{"id":"composability","label":"Composability","weight":0.16,"why":"insufficient evidence","sources":[],"evidence":"insufficient"},{"id":"checklist","label":"Checklist","weight":0.1,"value":0,"contribution":0,"why":"No MCP surface is declared. No API is declared. Cost is unknown. 0 of 3 checklist fields are complete.","sources":[],"evidence":"observed"},{"id":"evidence_quality","label":"Evidence quality","weight":0.14,"value":0.18107144632819594,"contribution":0.025350002485947435,"why":"1 sources cover 1 evidence types. The latest dated evidence is 2026-10-19. Freshness is omitted unless observed as a signal.","sources":[{"kind":"web","label":"executor","url":"https://github.com/eduagarcia/lm-evaluation-harness-pt","excerpt":"BranchesTags\n\n## Latest commit\n\n## History\n\n3,279 Commits\n\n## Folders and files\n\n## Repository files navigation\n\n# Language Model Evaluation Harness for Portuguese LLMs\n\n**A framework for evaluating Large Language Models (LLMs) in Portugues","at":"2026-10-19T01:26:54.232Z"}],"evidence":"observed"},{"id":"benchmark","label":"Benchmarks","weight":0.12,"why":"insufficient evidence","sources":[],"evidence":"insufficient"}],"missing":["GitHub health: insufficient evidence","Agent use: insufficient evidence","Human notes: insufficient evidence","Composability: insufficient evidence","Benchmarks: insufficient evidence"]},"score":3}