Plotly's November benchmark puts an open-weight Kimi model within one question of the top score. The result shows how analytics agents can be compared on shared tools and data, and why ambiguity, repeated testing and deployment costs matter
A chart can be accurate and still answer the wrong question. Consider a hypothetical wind fleet whose operations team wants to know which site performed best. The site that generated the most electricity might be the largest. A smaller site might have made better use of its installed capacity, and a third might have had fewer interruptions. Each answer could follow from a valid database query. The analytical work begins with deciding which comparison makes sense, identifying the records needed to support it and recognising when those records leave the question unresolved.
On 6 November 2026, Plotly reported that Moonshot AI’s Kimi K3.1 Pro scored 42 out of 43 on its DataAnalyticsBench evaluation. Two proprietary models scored 43. Plotly used the same agent tools and a synthetic wind-energy warehouse with 20 million rows. Moonshot’s repository makes the model files available under a named Kimi licence. The new result is a comparison of analysis systems in one controlled setting, with a different purpose from evaluating the capabilities of the earlier Kimi K3 on general tasks. Plotly’s November comparison and Moonshot’s model repository.
The comparison matters because an analytics agent operates at the intersection of language, software and evidence. The model decides which action to request, while the surrounding software provides the database connection and executes permitted operations. The database can compute a result precisely from the instructions it receives. A good agent must also decide whether those instructions preserve the meaning of the original question. A single score compresses several distinct capabilities, from finding a relevant table to explaining the limits of a result. The practical significance of a mistake depends on which capability failed.
When a successful query is the easy part
Plotly’s methodology includes questions about ambiguous requests, unavailable evidence, statistical pitfalls and data defects, alongside calculations with expected answers. Its rubric can give credit for identifying an ambiguity, including when an agent states its assumption. The benchmark also includes patterns planted in the data that the agent must discover. These task types put different demands on the system. An agent can be good at translating a precise instruction into a query and struggle when the instruction leaves the denominator or time period open. DataAnalyticsBench methods and example questions.
Imagine an analyst asking how many hours a hypothetical turbine fleet spent under alarm. A database may store an individual record for each alarm, with its starting time and duration. Adding those durations could count an hour twice if alarms overlapped. Merging the overlapping intervals would answer a question about elapsed time with any alarm active. The summed durations could still be relevant to another question about the volume of alerts. The model needs to understand which quantity the person is seeking, then use the database to calculate it. The query engine has no independent way to resolve that choice.
Earlier work on enterprise text-to-SQL illustrates a related challenge. The authors of Spider 2.0 assembled 632 workflow problems derived from enterprise database use cases. Their evaluation encompasses different SQL dialects and operations that can require multiple queries. That research provides context for why examining a schema and planning transformations are worth testing. It does not replicate Plotly’s scores. An agent whose results look strong in one environment may behave differently when documentation is incomplete or a different database expresses the same operation in unfamiliar terms. Spider 2.0 research paper.
Read the score with its uncertainty attached
Plotly says it ran each model once. The questions were in English, used a single warehouse and ran through Plotly Studio Embedded. A model graded final answers against the key or checked whether a response recognised the issue built into a question. The authors describe the rankings as coarse and explicitly caution against reading much into a difference of one or two answers. That is particularly relevant when comparing 42 with 43. The test also leaves open how an agent behaves across a longer conversation, where early assumptions can carry into subsequent requests. Plotly’s benchmark limitations.
Repeated testing would separate a plausible pattern from the result of a particular run. In a proposed follow-up, evaluators could rerun the same tasks with identical settings, then repeat them with paraphrased requests and unseen tables. It would also be useful to review disagreements between the automatic grader and an analyst who can inspect the queries and evidence. A concise answer might be marked as missing a caveat even when the underlying work identifies it. A persuasive answer might satisfy a superficial check while relying on an incorrect join. Inspecting the path to the answer helps distinguish those cases.
Synthetic data allow benchmark designers to plant a defect and know where to look for it. Production data can be harder to interpret because nobody may know in advance which measurements are missing or why an apparent discrepancy exists. A change in sensor reporting can resemble a shift in performance. A table can be complete for one site and partial for another. Those possibilities help explain why transfer to a new domain requires new evidence. The published comparison makes visible which behaviours the designers value, while the team deploying a system still has to define what counts as a sound answer in its own setting.
Available weights change the deployment question
Moonshot publishes model files and instructions for using Kimi K3.1 Pro, with a licence identified on the repository. That availability gives a team the option to inspect the terms and consider running a compatible version under its own control. The decision still depends on the hardware, software and people needed to serve it. In particular, a downloadable model need not fit on an analyst’s workstation. A team that deploys a compressed version or changes the runtime should verify its results again, since the published comparison describes a particular hosted setup. Kimi K3.1 Pro model card and files.
Plotly reported a cost of $13.47 for the Kimi run and a median of around two and a half minutes per question. It reported $7.85 and 38 seconds for a hosted Grok 4.7 run with the same score. Those figures reflect the tested providers and their pricing conditions. They say little about the cost of operating the model internally or the time an analyst would spend checking its conclusions. For an interactive dashboard, waiting for a query can be the dominant constraint. A scheduled overnight analysis can instead prioritise coverage and the time needed to resolve mistakes. Plotly’s cost and latency comparison.
A technically meaningful comparison would account for the rest of the system. Some questions may generate many database calls, which use warehouse capacity even if the model itself is inexpensive. A slow query can hold up another user. Meanwhile a quick answer can be costly if a human has to reconstruct the logic before trusting it. Self-hosting changes where these costs land and who must monitor the service. It also changes the location of model execution, while security still depends on database permissions, the treatment of logs and the handling of outputs. Those are system properties that an accuracy leaderboard cannot settle.
A benchmark for the analyst’s job
One promising application is exploratory work in a restricted workspace, where an analyst can see what the system queried and compare the result with established reporting. Consider a hypothetical maintenance team trying to explain a rise in service calls. An agent might propose a few comparisons and point to an apparent concentration in one category. The analyst could examine the query and check whether the number of machines in that category changed at the same time. In this arrangement the agent can reduce the effort of looking for a pattern, while the team retains the ability to decide what the observation warrants.
A stronger evaluation would reserve particular attention for questions the data cannot answer. In real work, a confident guess can move into a report and become difficult to unwind once it shapes a decision. A well-calibrated response would identify the missing evidence or ask which definition the requester intends. The testing team could grade that behaviour explicitly, check whether the answer remains stable when phrased differently and measure how often a human has to intervene. Keeping the resulting query and its assumptions with the answer would make the system easier to investigate when a later result contradicts an earlier one.
The November comparison shows that an available-weight model came close to the top score in a specified analytics environment. The most interesting next comparison would use the data, permissions and questions of the people who need the system, with repeat runs and a record of how errors are caught. That would make it possible to decide whether the system can support exploration, draft a recurring report or handle only a narrower class of queries. As those choices become more consequential, an agent earns trust through the evidence it leaves behind and through the occasions when it can explain why an answer is still out of reach.
Image credit: Wind turbines DK by Lowcarb23, CC BY-SA 4.0. Illustrative photograph. Plotly’s test fleet is fictional. Resized for display.
