{"id":439153,"date":"2024-02-27T15:56:00","date_gmt":"2024-02-27T14:56:00","guid":{"rendered":"https:\/\/blog.jetbrains.com\/?post_type=dotnet&p=439153"},"modified":"2024-03-19T17:58:52","modified_gmt":"2024-03-19T16:58:52","slug":"critical-thinking-in-an-ai-powered-world","status":"publish","type":"dotnet","link":"https:\/\/blog.jetbrains.com\/pt-br\/dotnet\/2024\/02\/27\/critical-thinking-in-an-ai-powered-world","title":{"rendered":"Critical Thinking in an AI-Powered World"},"content":{"rendered":"\n

Last year, we launched JetBrains AI Assistant, and from JetBrains’ perspective, our mission of improving your drive to develop hasn\u2019t changed over the past two decades. As we head into 2024, Artificial Intelligence (AI) will likely be the most talked about topic of the year. Wherever you fall on the spectrum of AI conversation, one thing is clear: we must all equip ourselves with new critical thinking skills. <\/p>\n\n\n\n

In this post, we\u2019ll talk about the technology that powers JetBrains AI Assistant<\/a>, its strengths and weaknesses, examples of edge cases, and strategies to get better results from the JetBrains AI Assistant.<\/p>\n\n\n\n

What are LLMs? Strengths and Weaknesses<\/h2>\n\n\n\n

Large Language Models (LLMs) are the technology powering many AI services that require natural language processing. These models are typically trained to achieve general-purpose language tasks and have been described as \u201cautocorrect on steroids\u201d by many. LLMs are a complex web of decisions known as a neural network. Most LLMs have settled on a chat interface with a feedback loop designed to refine a particular task set by the user further.<\/p>\n\n\n\n

Models typically have three distinguishing factors: Tokens, number of parameters, and training dataset cutoff dates. <\/p>\n\n\n\n

The tokens define how much context a user can define in a session. What a token is<\/em> depends on the LLM, as it is a representation of input. A token could be a word, a character, or a meaningful data representation. With access to more tokens, an LLM can have more context to complete a user\u2019s task, although more isn’t necessarily better in the case of tokens.<\/p>\n\n\n\n

Parameters are LLMs’ second distinguishing, and not always public, feature. You can think of parameters as decisions along the neural network. The more parameters, the more significant the number of decisions an LLM model may exercise to produce an answer. Current models have parameters in the Billions,<\/strong> with future models predicted to be in the Trillions.<\/strong> Again, more isn\u2019t always necessarily better, as a smaller model trained for a specific use case may outperform a more extensive model on task results and time taken to respond.<\/p>\n\n\n\n

Finally, and most important to the user experience, is the training cutoff date. Many of these models use public datasets to train. These cut-off dates become more critical in tasks relating to time-sensitive knowledge, such as current events and cutting-edge programming techniques. Cutoff dates are not a clear indication of quality, as some model providers still keep much of their datasets secret due to the potential use of copyrighted material.<\/p>\n\n\n\n

JetBrains developer advocate Jodie Burchell<\/a> is an expert in data science and has broken down the strengths and weaknesses of LLMs.<\/p>\n\n\n\n

Strengths<\/strong><\/th>Weaknesses<\/strong><\/th><\/tr><\/thead>
Summarization<\/td>Maths<\/td><\/tr>
Text & Code Generation<\/td>Randomness<\/td><\/tr>
Classification<\/td>Timeliness<\/td><\/tr>
<\/td>Non-deterministic <\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

We\u2019ll explore some of these in the following sections and strategies to counteract them in your attempt to find solutions.<\/p>\n\n\n\n

The Temporal Disconnect<\/h2>\n\n\n\n

As a technologist, you likely understand the break-neck speed at which all ecosystems are currently moving. Whether you stay current with the latest trends is a personal choice, but if you\u2019re an early adopter, you may have noticed AI-generated code samples can sometimes feel \u201cdated\u201d.<\/p>\n\n\n\n

For example, at the time of writing, ChatGPT has a training dataset up to September 2021<\/a>. The cutoff date means the embedded information in the ChatGPT model is unaware of any Java version after 16 and is undoubtedly unaware of the latest C# 12 features in .NET.<\/p>\n\n\n\n

Let\u2019s take a look at a .NET example specifically regarding testing dates.<\/p>\n\n\n\n

[Fact]\npublic void Ladies_An_Gentlemen_The_Weekend()\n{\n      var now = DateTime.UtcNow.DayOfWeek;\n            \n      Assert.Contains(now, \n          [DayOfWeek.Friday, DayOfWeek.Saturday, DayOfWeek.Sunday]\n      );\n}<\/pre>\n\n\n\n
Regardless of your technology stack, you likely notice an immediate issue. This test will pass intermittently based on the current date and time. There are multiple ways to solve this issue, but let\u2019s ask the JetBrains AI Assistant first.<\/p>\n\n\n\n
Suggest a way to refactor the variable `now` so that I can control the value without depending on `DateTime.UtcNow`<\/pre>\n\n\n\n
The response provides a custom IDateProvider<\/code> implementation. The proposed functionality is a correct solution, but a developer with expertise in the ecosystem might question whether it\u2019s the only solution.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
.NET 8 recently added a TimeProvider<\/code><\/a> class, so let\u2019s ask JetBrains AI Assistant to refactor to use the mechanisms built into the SDK.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
The unexpected result can be a frustrating experience for those who know a feature exists but can\u2019t seem to get the JetBrains AI Assistant to generate the expected code. Let\u2019s write the TimeProvider<\/code> code ourselves.<\/p>\n\n\n\n
private readonly FakeTimeProvider timeProvider = new();\n\n[Fact]\npublic void Ladies_An_Gentlemen_The_Weekend()\n{\n     timeProvider.SetUtcNow(new DateTime(2021, 1, 8)); \/\/ Friday\n     var now = timeProvider.GetUtcNow().DayOfWeek;\n     Assert.Contains(now, \n       [DayOfWeek.Friday, DayOfWeek.Saturday, DayOfWeek.Sunday]\n     );\n}<\/pre>\n\n\n\n
While the chat may not yield the initial expected results, as we begin to write another test, the JetBrains AI Assistant becomes aware of usages and patterns around how we may want to write some of our subsequent tests.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Remember, LLMs are excellent at predicting text, and while newer knowledge of language features may not exist within the model, most newer language features behave similarly to older ones. The patterns in languages can aid JetBrains AI Assistant, and your code can help your sessions be productive by providing additional context.<\/p>\n\n\n\n
If you find a temporal piece of information commonly missing from your sessions, you can create a new prompt to describe your expected behavior. Remember, LLMs are excellent at predicting future text from the previous examples.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Custom prompts can be a great way to overcome the limitations of any model until providers have updated the training data and contain newer embedded information.<\/p>\n\n\n\n
Strategies for Better JetBrains AI Assistant Responses<\/h2>\n\n\n\n
Like most things, the more effort you put into JetBrains AI Assistant sessions, the better your results. Previously, we had mentioned LLMs aren\u2019t particularly strong with math.<\/p>\n\n\n\n
Let\u2019s work with JetBrains AI Assistant to generate a method that calculates the time it takes for an object to fall from a certain height in meters using the gravitational constant of Earth\u2019s gravity. We\u2019ll use the following code and let code completion fill out the rest.<\/p>\n\n\n\n
using System;\n\n\/\/ Method `CalculateFallTimeAndVelocity` calculates\n\/\/ the time in seconds an object takes\n\/\/ to fall to the ground based on its height\n\/\/ in meters returns time and velocity as a Tuple\n(double time, double velocity) CalculateFallTimeAndVelocity(double meters)\n{\n    \n}\n\nvar (time, velocity) = CalculateFallTimeAndVelocity(100);\nConsole.WriteLine($\"time: {time}, velocity: {velocity} m\/s\");\n<\/pre>\n\n\n\n
In the method’s body, JetBrains AI Assistant recommends implementing known formulas.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
This solution is good, but you should immediately become skeptical whenever you see numbers in a response.<\/p>\n\n\n\n
Ask yourself the following questions:<\/p>\n\n\n\n