Understanding ambiguity limitations with large language models
Like humans, large language models do not handle ambiguity well. The responses to ambigous unguided prompts are often high level and summarized. This is where the huamn driven factor of AI driven development is extremely valuable.
GPT cannot generate fully functioning or integrated applications in a single request. For example, you cannot simply tell GPT to create a bicycle rental application and expect a fully functioning application as an output.
Type it in as a prompt and see what happens: “Create a bicycling rental web application where users can rent a bike”.
GPT provides the following response:
Define the scope of the application: What kind of bikes will be offered for rent? What are the rental rates? What are the rental durations? Will the application offer any discounts or promotions? These are just a few of the questions that need to be answered when defining the scope of the application.
Develop a user interface (UI) for the application: The UI should be designed in a way that makes it easy for users to browse bikes, select rental options, and make payments.
Implement a bike inventory system: The application needs to keep track of the number of bikes available for rent, their location, and their rental status.
Implement a payment system: The application should offer a secure payment gateway that allows users to pay for their bike rentals using credit cards, PayPal, or other payment methods.
Implement a reservation system: Users should be able to reserve bikes in advance and receive confirmation of their reservation via email or SMS.
Implement a user management system: The application should allow users to create and manage their profiles, view their rental history, and update their payment information.
Implement a feedback system: The application should allow users to provide feedback on their rental experience, which can be used to improve the service in the future.
Launch and market the application: Once the application is ready, it can be launched and marketed through social media, email campaigns, and other channels to attract users.
Monitor and optimize the application: It's important to monitor the application's performance regularly and make improvements as needed to ensure that users have a positive experience.
These are just a few of the steps involved in creating a bicycling rental web application. Depending on the specific requirements of the application, additional steps may be necessary. It's also important to work with a team of experienced developers who can help bring your vision to life.
While the above response is useful and gives us features and systems to consider, it does not output any code.
Understanding scope of truth related to generative AI
Most large language models are typically not trained to lie. They simply evaluate the prompt against its model (the data it was trained on), and return a response. These models are updated at a point and time, meaning they may have been built on data that is days, months, or years old. And while responses can sound very confident, AI can be incorrect through its misunderstanding of context, application or simply be out of date.
While this applies to all questions, this is especially true in software development due to the fast pace of change within languages, libraries, and platforms.
Another limitation to LLMs is the scope of it's knowledge. Let's say there is an updated charting library being using in existing development framework like React. A particular exception and it's solution may not be something that the model has been trained on and may not be able to answer that question. Or if a certain method in a framework was never documented publicly, the LLM may respond that the method does not exist when it really does.
No matter how rapidly AI is refined, there will always be the possibility for AI to create code that results in exceptions that it cannot solve. I have encountered a few situations where GPT is continually generating clean code that compiles without issue, and suddenly gets caught on something that it cannot resolve. Have patience and run through a structured troubleshooting process. A process for troubleshooting is provided at the end of this course, as a resource for anytime you run into an issue while developing. To be successful, have the expectation that AI is a tool to build applications and cannot be used an “app generator” at this point and time.
Checking when data on model on was last updated will like vary per model.
For ChatGPT simply ask it as a prompt.
This model's data was last updated in September 2021.
Using generative large language models for visualization tasks
ChatGPT cannot output visual content. Its understanding of visual elements come from textual data describing different visual elements such as navigation bars, hero sections, or tables. While this can be effective in building good looking UI elements, you should expect some limitations in this domain. We will go through various visualization use cases in detail in the Visualizatn domain.
Comparing that to a dedicated ASCII art generator:
Even when prompted for a more detailed version, ChatGPT's visualization capabilities are currently limited.
This ASCII art includes more detail, such as feathers, claws, and a beak. ASCII art can range from simple designs to highly detailed and intricate artwork. It's a fun and creative way to express oneself using only the keyboard.
And when we ask GPT-4, we get a more honest response that it is limited in it's ability to generate ASCII art, but the art provided is more accurate.
Unfortunately, creating an extremely detailed owl using ASCII characters is beyond the scope of what I can provide within this text format. However, I can provide you with a simpler ASCII owl and a caption for it. You can always look for more detailed ASCII art online or use an ASCII art generator tool to create one.
Here's a simple owl in a markdown code block with a caption:
Limitations of generative AI in developing applications
The limitations in this section can be generally applied to all LLMs.
The following limitations may be found in various generative tools:
Ambiguity
Truth
Time Accuracy
Scope
Visualizations
Visualization limitations are specific to large language models in generative AI. However, large language models can be augmented with machine learning image analysis or generative image tooling. For example, As of GPT-4, the platform supporst the ability to consume and understand images and output responses from it.
As ChatGPT is this most capable and accessible in the LLM space, this course focuses on ChatGPT so we will specifically discuss the limitations and weaknesses of this platform, which may or may not be the same as other platforms.
AI can be wrong about programming concepts. In the same way GPT is not completely accurate with all code generation, we should be critical of the code generated and how it is implemented. Always thoroughly review them for accuracy and cognitive complexity. This will help build competency up in various patterns and concepts.
These limitations are far from prohibitive, but are something that must be considered during the development process. Having a good understanding of these limitations provides developers focus areas and context that make the development process and application more efficient.