Performance of Large Language Models (LLMs) in Complex Analysis: A Benchmark of Mathematical Competence and its Role in Decision Making.

By Jaime Esteban Montenegro Barón @ 2025-05-06T21:08 (+1)

+6

5By: Jaime Esteban Montenegro.

This project was conducted as part of the "Careers with Impact" program during the 14-week mentoring. You can find more information about the program in this post.

Contextualization of the problem.

The accuracy of Large Language Models (LLMs) in solving advanced mathematical problems in Spanish and their impact on decision making in fields such as economics, medicine and education are key questions today. It is important to consider here the accuracy in Spanish, being the second most spoken mother tongue globally and with its own linguistic structures (What is the most spoken language? 2025), whose correct interpretation by LLMs-trained predominantly in English-is necessary for reliability in Spanish-speaking contexts. On the other hand, although various users trust the results generated by these technological tools, they often do not know if they are correct. Confidence in AI is often based on its apparent sophistication, without a critical evaluation of the accuracy of its answers.

Although AI is based on mathematical principles, LLMs do not replicate human deductive reasoning, but rather identify patterns in textual data. This method, useful for routine problems, limits its reliability in complex tasks that require rigorous logic, generating errors or "hallucinations." Benchmarks such as Frontier Math (Glazer, et al., 2024) assess general mathematical abilities of LLMs, but advanced areas such as Complex Variable Functions represent only 2.4% of the total, compared to domain predominant such as Theory of Numbers (17.8%) or Combinatorics (15.8%). Even compared to specialized branches such as Differential Geometry (1.4%) (Glazer, et al., 2024), its low participation shows a gap in the evaluation of technical capabilities, which reinforces the need for dedicated studies to understand their performance in specialized domains.

Understanding the limitations of LLMs in under-evaluated areas, such as complex analytics, is critical for informed decisions in fields such as medicine, computer security, or engineering because of the type of consequences that such decisions encompass. Given the frequent application of LLMs in data analytics, predictive modeling, and automation of complex tasks, an error on the part of the model, stemming from its inability to rigorously handle complex concepts, can have serious repercussions on the outcome of a decision. Therefore, assessing their accuracy in scenarios where accuracy plays an important role is indispensable to establishing their viability as decision support tools.

Research Question

What is the performance of Large Language Models (LLMs) when faced with problems in specialized areas of advanced mathematics, such as complex analysis, and how does this impact decision making in the other areas of knowledge?

General Objective

Develop a benchmark to evaluate the advanced mathematical reasoning of ChatGPT-4 and DeepSeek in the domain of complex analysis, using problems in Spanish that require a solid foundation in mathematics, and the Zero-Shot prompting technique to generate specific results.

Specific Objectives

1. Collect and select a diverse set of 25 complex analysis problems.
2. Adapt the Zero-Shot1 prompting technique for each selected mathematical problem.
3. Run the benchmark tests with the ChatGPT-4 and Deepseek LLMs.
4. Statistically analyze the results obtained to identify the strengths and weaknesses of LLMs in each mathematical area under the Zero-Shot technique.

Personal Objectives

1. To use my background in mathematics to identify and adapt problems that constitute effective tests of mathematical reasoning, and to develop my ability to accurately diagnose AI strengths and weaknesses based on my understanding of the methods and subtleties of complex analysis.
2. Master the adaptation and application of prompting techniques in formal reasoning contexts.
3. Acquire experience in the design and systematic execution of Benchmarks.

1 Prompt Zero-Shot: The prompt used to interact with the model does not contain examples or demonstrations (Zero- Shot Prompting| Prompt Engineering Guide, 2025).

METHODOLOGY

Figure 1. flow diagram of the complex analysis benchmark methodology. The figure illustrates the methodological flujo de trabajo employed in the study to evaluate the capabilities of LLMs in the complex analysis domain under the Zero- Shot prompting technique.

The project was developed in the following stages:

1. Definition of problems: a set of complex analysis problems of Advanced difficulty University Level / Initial Postgraduate), requiring the application of key concepts and theorems, are defined. These problems had to meet the following criteria:
2. Thematic and conceptual relevance: The selected problems from a relevant set because of their representativeness of the fundamental analytical and problem- solving capabilities that define the domain of complex analysis. The benchmark is designed to assess the models' ability to understand and apply the essential theoretical framework and key operational techniques used in the discipline, covering the types of problems that are central to the mastery of its main theorems and methods. This relevance ensures that the measured performance reflects mastery of the basic computational and conceptual skills required to tackle analytical tasks in this mathematical field.
3. Complexity and Diversity of the Problem Set: The problem set has been selected to constitute a benchmark representative of the complexity and breadth of advanced mathematical analysis. The selected problems require more than the direct application of elementary definitions or formulas. They require the thorough understanding, proper selection, and integrated application of fundamental theorems and techniques of complex analysis, such as (but not limited to) Cauchy's Integral Theorem and its extensions, the Residue Theorem (applied to different types of singularities and contexts, including the evaluation of real integrals by contours), the handling of multivalued functions, and principles related to harmonic functions and conformal transformations. Included are problems that demand multi-step chains of reasoning, analysis of conditions (such as the orientation of the contour or the main branch of a function), and the ability to deal with non-trivial situations or with applications (such as contour value problems).
4. Clarity: The problems have been selected and adapted to ensure an unambiguous and complete formulation, minimizing any potential source of interpretive ambiguity on the part of the language models. Additionally,

2 Difficulty level (Advanced): Requires a solid command of the theory of complex variable functions and application of key results in complex integration, singularity analysis, holomorphic functions and contour problems.

The wording was concise and direct, avoiding constructions that could generate semantic or contextual confusion, thus ensuring that the assessment focuses purely on mathematical reasoning ability.

1. Originality: We sought to include original and novel problems that the LLM finds unusual, to assess the advanced mathematical reasoning capabilities of the models beyond the mere retrieval of memorized solutions or pre-existing answer patterns.

The set of problems with their respective solutions can be seen here.

1. Benchmark design: A benchmark was constructed that includes the Zero-Shot prompting technique that assesses problem-solving ability with pre-trained LLM knowledge, without examples or hints.

Experimental design approach

1. Exact Match (EM) and Logical Validity metrics were established to evaluate LLMs' performance, such as problem-solving accuracy and response consistency.
2. A controlled experimental design was used to compare the performance of different LLMs on the benchmark.
3. Statistical tests such as McNemar's test 3 and Wilcoxon's test4 for signed ranks were performed to determine whether differences in model performance are significant.
4. Selection of LLMs: The state-of-the-art language models ChatGPT-4 and Deepseek were selected.
5. Implementation and evaluation: The benchmark was implemented and the performance of the LLMs in problem solving was evaluated, taking into account that each of the problems was executed on the official websites of each LLM. In addition, the results were compared with the solutions generated by mathematicians.

3 McNemar test: This is a statistical test used to compare two related (paired) proportions when the data come from the same subjects or items under two different conditions.

4 Wilcoxon Signed-Rank Test: A nonparametric test designed to compare two related (paired) measurements when the data are numerical and do not necessarily meet the assumption of normality of differences.

1. Analysis of results: The analysis of results, which integrated quantitative and qualitative approaches, sought to identify the strengths, weaknesses and limitations in the advanced mathematical reasoning of the evaluated models. Quantitatively, the performance of each experimental condition was determined by calculating the proportion of correctly solved problems and a metric of the correctness of the solving steps; significant differences between conditions in this paired design were established using McNemar (for proportions) and Wilcoxon for Signed Ranks (for the steps metric) statistical tests, applying the relevant corrections for multiple comparisons and relying on visualizations. Qualitatively, the content of the generated textual responses was analyzed, including the systematic coding of the observed errors according to their type (conceptual, procedural, computational, interpretation, etc.) to identify patterns and underlying causes, as well as the examination of the sequences of steps and justifications.

Results and Discussion

What problems did Deepseek and ChatGPT-4 solve correctly and incorrectly under Prompt zero-shot?

Table 1. Comparison showing which problems ChatGPT-4 and Deepseek answered correctly.

How many problems did Deepseek and ChatGPT-4 answer correctly and incorrectly?

Figure 2. Comparison of the overall performance of Deepseek and ChatGPT-4. The figure illustrates the number (left graph) and percentage (right graph) of benchmark problems answered correctly and incorrectly by Deepseek and ChatGPT-4 under the Zero-Shot prompting technique. Deepseek obtained a slightly higher percentage of correct answers (68%) than ChatGPT-4 (60%) on this set of problems.

How many correct steps did Deepseek and ChatGPT-4 perform per problem?

Table 2. Comparison showing the number of steps vs. the number of correct steps per problem solved by Deepseek.

Figure 3. Detailed analysis of Deepseek's solving process under the Zero Shot prompting technique. The figure presents, for each benchmark problem (horizontal axis), the total number of steps attempted by Deepseek (blue bar) compared to the number of correct steps within that attempt (green bar). This graph allows us to evaluate the extent and accuracy of the model solving process at the individual level per problem.

Figure 3 allows to visually identify problems (such as PR1, PR8, PR17, PR20) where the number of correct steps is notably lower than the total number of steps, indicating partial solution attempts or with significant errors despite having generated several steps.

Table 3. Error Probability of DeepSeek by Number of Steps

To examine the general trend, problems are grouped into two categories: problems with 5 or fewer steps and problems with more than 5 steps.

• Problems with 5 steps or less: There are 19 problems in this category. Six errors were observed. The probability of errors in this group is 26.30%.
• Problems with more than 5 steps: There are 6 problems in this category. Four errors were observed. The probability of error in this group is 66.7%.

Pearson's correlation coefficient: 0.47.

Type of correlation: Moderate positive linear.

Conclusion about Pearson's correlation coefficient

There is a tendency for problems with more steps to have a higher probability of error, but the relationship is not extremely strong.

5 Pearson's correlation coefficient: A test that measures the statistical relationship between two continuous variables (What Is Pearson's Correlation Coefficient, n.d.).

Table 4. Comparative table showing the number of steps vs. the number of correct steps for each problem solved by ChatGPT-4.

Figure 4. Detailed analysis of the ChatGPT-4 solving process under the Zero-Shot prompting technique. The figure shows, for each benchmark problem (horizontal axis), the total number of steps attempted by ChatGPT-4 (blue bar) in Zero-Shot condition compared to the number of correct steps within that attempt (green bar). This graph allows us to evaluate the extent and accuracy of the ChatGPT-4 solving process at the individual per-problem level.

In figure 4 we observe problems (such as PR8, PR19, PR20) where ChatGPT-4 generates a relatively high number of total steps (high blue bar), but the number of correct steps is comparatively very low (very short green bar). This indicates that, for those problems, the model attempted an extensive resolution process, but this process had a considerable number of errors, conceptual or computational detours, or simply got "lost" halfway through without managing to correctly execute most of the necessary steps.

Table 5. ChatGPT-4 Error Probability by Number of Steps.

To examine the general trend, we do the same as we did for Deepseek; we group the problems into two categories: problems with 5 or fewer steps and problems with more than 5 steps.

• Problems with 5 steps or less: There are 20 problems in this category. Seven errors were observed. The probability of errors in this group is 35%.
• Problems with more than 5 steps: There are 5 problems in this category. Three errors were observed. The probability in this group is 60%.

Pearson's correlation coefficient: 0.235

Type of linear correlation: weak positive

Conclusion about Pearson's correlation coefficient

There is a slight tendency for problems with more steps to have a higher probability of error, but this linear relationship is weak in this data set.

Figure 5. Comparison of the total number of steps per problem between Deepseek and ChatGPT-4 under the Zero Shot prompting technique. The figure shows, for each benchmark problem (horizontal axis), the total number of steps attempted by Deepseek (blue bar) versus the total number of steps attempted by ChatGPT-4 (pink bar). This graph

facilitates direct comparison of the extent of the resolution process between the two models for each individual problem.

In figure 5, specific problems are evident where the process length differs markedly: for example, in PR16, Deepseek uses 5 steps while ChatGPT-4 uses only 2; in PR11, Deepseek uses 3 steps while ChatGPT-4 uses 6; in PR20, Deepseek uses 6 steps versus ChatGPT-4's 8.

Figure 6. Comparison of the percentage of correct steps per problem between Deepseek and ChatGPT-4 under the Zero Shot prompting technique. The figure shows, for each benchmark problem (horizontal axis), the percentage of correct steps achieved by Deepseek (blue bar) versus the percentage of correct steps achieved by ChatGPT-4 (pink bar) in the Zero- Shot condition. This graph facilitates direct comparison of the relative quality of the solving process between the two models for each individual problem.

Looking at the pairs of bars in figure 6, we notice that for a considerable number of problems (especially those where both were successes, as seen in Figure 1), both models achieve 100% correct steps (bars at the maximum of the scale), indicating a flawless execution of the process they attempted. However, in problems where one of the models did not achieve 100%, differences in process quality become visible; for example, in PR1, Deepseek achieved 66.67% correct steps versus 33.33% for ChatGPT-4; in PR3, Deepseek achieved 100% while ChatGPT-4 achieved 0%; in PR17, Deepseek achieved 50% versus 62.5% for ChatGPT-4.

McNemar Test

Table 6. Contingency table for McNemar test

• Null Hypothesis ()6: There is no significant difference in the proportion of

correctly solved problems between ChatGPT-4 (Zero-Shot) and DeepSeek (Zero- Shot). (The probability that GPT-4 hits and DeepSeek misses is equal to the probability that GPT-4 misses and DeepSeek hits).

• Alternative Hypothesis ()7: There is a significant difference in the proportion of

correctly solved problems between the two conditions (the above probabilities are not equal).

Chi-Square statistic with Yates correction8:

• The statistic follows a χ² distribution with 1 degree of freedom.
• The p-value associated with= 0.5 is 0.4795.

Since the p-value is greater than the common significance level (α = 0. 05), we cannot reject the null hypothesis.

McNemar Test Conclusion

There is no significant difference between ChatGPT-4 and DeepSeek in their correct/incorrect classifications under the zero-shot approach (α= 0.05) to conclude that there is a difference in the proportion of correctly solved problems between ChatGPT-4 (Zero-Shot) and DeepSeek (Zero-Shot) on this set of 25 problems. The small difference in discordant results (DeepSeek ZS got 2 rights that GPT-4 ZS missed, while GPT-4 ZS got none rights that DeepSeek ZS missed) is consistent with being random variation.

Wilcoxon signed-rank test (PERCENTAGE OF CORRECT STEPS)

6 Null Hypothesis (H0): The null hypothesis is usually an initial affirmation that is based on previous analyses or specialized knowledge. (About The Null and Alternative Hypothesis - Minitab, n.d.).

7 Alternative Hypothesis (H1): The alternative hypothesis is what you might think is true or expect to prove to be true. (About The Null and Alternative Hypothesis - Minitab, n.d.).

8 Chi-Square with Yates correction: This is the correction made when the expected frequencies in the calculation of the chi-square statistics are small, so that 0.5 is subtracted from each difference between expected and observed frequencies (Sarasola, 2024).

• Null Hypothesis (): There is no systematic difference in the distribution of the percentage of correct steps between DeepSeek and ChatGPT-4 with the Zero-Shot prompting technique. (The median of the differences is zero).
• Alternative Hypothesis (): There is a systematic difference in the distribution of the percentage of correct steps between the two models (The median of the differences is not zero).

Table 7. Table of differences and signed ranks for the Wilcoxon test (percentage of correct steps).

Ranges of non-zero differences (ordered by absolute value):

1. 2.38 (PR8) - Rank 1
2. 8.33 (PR20) - Rank 2
3. 12.50 (PR17) - Range 3
4. 17.14 (PR19) - Rank 4
5. 30.00 (PR24) - Range 5
6. 33.34 (PR1) - Rank 6
7. 100,00 (PR3) - Tie
8. 100.00 (PR16) - Tie. The tied ranks are in positions 7 and 8. The average rank is (7 + 8)/2= 7. 5.

Assignment of Sign to Ranks:

• PR1 (+33.34): +6
• PR3 (+100.00): +7.5
• PR8 (-2.38): -1
• PR16 (+100.00): +7.5
• PR17 (-12.50): -3
• PR19 (-17.14): -4
• PR20 (+8.33): +2
• PR24 (-30.00): -5

Sum of Signed Ranks:

• Sum of Positive Ranks  
• Sum of Negative Ranks ()

Wilcoxon test statistics (W):

For N=8 (number of non-zero differences) and a statistic W=13 we calculate the value of

𝑝 for a bilateral test.

Since the p-value is greater than the common significance level (α= 0. 05), not, we reject the null hypothesis.

Conclusion of Wilcoxon test with signed ranks for percentage of correct steps

There is no statistically significant evidence (at a significance level of 0.05) to conclude that there is a systematic difference in the distribution of the percentage of correct steps between Deepseek and ChatGPT-4 with the Zero-Shot prompting technique on this set of 25 problems. In terms of the quality of the solving process as measured by this metric, the data does not show a conclusive difference between the two models under this prompting technique.

Wilcoxon Signed Ranks Test (NUMBER OF STEPS)

• Null Hypothesis (): The distribution of the number of steps is the same for both models (the median of the differences is zero).
• Alternative Hypothesis (): The distribution of the number of steps is different between models.

Table 8. Table of differences and signed ranks for the Wilcoxon test (number of steps)

Ranges of non-zero differences (ordered by absolute value):

1. Absolutes of difference= 1 (11 times): Ranks 1 to 11. Average rank= (
2. Absolutes of difference= 2 (5 times): Ranks 12 to 16. Average rank
3. Absolutes of difference= 3 (2 times): Ranks 17 to 18. Average Rank

Sum of Signed Ranks:

• Sum of positive ranks
• Sum of ranges negative

Wilcoxon test statistics (W):

For N=18 and W=63.5 (bilateral test), the p-value is approximately 0.338. Since the p- value is greater than the common significance level α=0.05, we do not reject the null hypothesis.

Conclusion of Wilcoxon test with signed ranks for number of steps

There is no statistically significant evidence (at a significance level of 0.05) to conclude that there is a systematic difference in the distribution of the length of the resolution process (measured by the number of steps) between DeepSeek and ChatGPT-4 under the Zero-Shot prompting technique in this set of 25 problems.

Wilcoxon test with signed ranks (NUMBER OF CORRECT STEPS)

• Null Hypothesis (): The distribution of the number of correct steps is the same for DeepSeek and ChatGPT-4 under the Zero-Shot prompting technique. (The median of the differences in the number of correct steps is zero).
• Alternative Hypothesis (): The distribution of the number of correct steps is different between the two models (The median of the differences is not zero).

Table 9. Table of differences and signed ranks for the Wilcoxon test (number of correct steps).

Ranges of non-zero differences (ordered by absolute value):

1. Absolutes of difference= 1 (10 times): Ranges 1 to 10. Average rank
2. Absolutes of difference= 2 (4 times): Ranks 11 to 14. Average rank=

(11+ 12+ 13+ 14)/4= 12. 5

1. Absolutes of difference= 3 (1 time): Rank 15.
2. Absolutes of difference= 5 (1 time): Rank 16.

Sum of Signed Ranks:

• Sum of positive ranges (
• Sum of ranges negative

Wilcoxon test statistics (W):

For N=16 and W=51 (bilateral test), the p-value is approximately 0.629. Since the p- value is greater than the common significance level α=0.05, we do not reject the null hypothesis.

Conclusion of Wilcoxon test with signed ranks for number of steps

There is no statistically significant evidence (at a significance level of 0.05) to conclude that there is a difference in the distribution of the raw number of correct steps per problem between DeepSeek and ChatGPT-4 under the Zero-Shot prompting technique on this set of 25 problems. Both models showed a similar number of correct steps overall, and the individual differences observed were not sufficiently consistent to indicate a systematic superiority of one model over the other in this metric under the Zero-Shot technique.

Conclusion Wilcoxon test with signed ranks in general

For the Zero-Shot prompting technique, all three analyses agree that there are no statistically significant differences between the two models evaluated.

Based on the comparison between ChatGPT-4 and DeepSeek under the Zero-Shot prompting technique, evaluated on 25 complex analysis problems using final outcome and process metrics, we draw conclusions with direct implications for decision making:

1. The results show that while both models achieved a favorable correct response rate (60% for ChatGPT-4 and 68% for Deepseek), their hit rates are far from the perfection needed for applications where absolute accuracy is required. With this, it is possible to conclude that even state-of-the-art LLMs in a structured domain are not inherently configured to generate consistently correct results.
2. The absence of statistically significant differences in key performance between these two specific models under Zero-Shot has a direct implication for deployment decision making. For tasks in any domain (scientific, technical, financial, etc.) requiring a level of reasoning analogous to that evaluated here and where the use of a Zero-Shot approach is considered, these results suggest that the choice between ChatGPT-4 and Deepseek will not provide a statistically reliable advantage in terms of accuracy or process quality.
3. The results of this benchmark serve as evidence that the confidence of LLMs on complex reasoning tasks is limited and not consistently superior among comparable models in simple configurations. This substantiates the critical decision that, in any decision-making process that relies on complex analysis or reasoning, the responses of these models should be treated as hypotheses, drafts, or preliminary analyses that require rigorous verification by human experts or independent validation systems.

Perspectives

In future work, it is suggested to increase the number of problems to make the statistical study with a more significant sample. However, the same process could be repeated with other prompting techniques such as Chain of thought or Few-Shot to measure their reliability. Nevertheless, one could correlate the performance of the models with objective characteristics of the problems (e.g., number of singularities, contour complexity, use of specific theorems) to identify which aspects of complex analysis problems are particularly challenging for LLMs.

Moreover, other leading-edge and emerging LLMs can be included in the benchmark to make a broader comparison of the current capabilities of the field.

REFERENCES

Glazer, E., Erdil, E., Besiroglu, T., Chicharro, D., Chen, E., Gunning, A., . . . Wildon, M. (December 20, 2024). FRONTIERMATH: A BENCHMARK FOR EVALUATING ADVANCED MATHEMATICAL        REASONING        IN        AI.        Arxiv,        26. Retrieved from https://arxiv.org/pdf/2411.04872

What is the most spoken language? (2025). Retrieved from Ethnologue: https://www.ethnologue.com/insights/most-spoken-language/

Zero-Shot Prompting (2025). Retrieved from Prompt Engineering Guide.

Sarasola, J. (2024). Yates correction - what is it, definition and frequently asked questions - Ikusmira. https://ikusmira.org/p/correccion-de-yates

About the null and alternative hypothesis - Minitab (n.d.). Retrieved April 23, 2025,        from

https://support.minitab.com/es-mx/minitab/help-and-how-to/statistics/basic- statistics/supporting-topics/basics/null-and-alternative-hypotheses/