How much "I" is in AI?

The news media love reporting on AI as if we are on the brink of machines taking over most traditional human jobs, such as writers, mechanics, doctors, lawyers, software developers, etc - any profession, really, that requires a good amount of knowledge, intuition and creativity. These headlines do catch attention and sell well, but how much intelligence really is there behind that blinking cursor of an AI prompt?

Under the hood

This section provides a high-level overview of how transformer models, such as GPT-4, generate responses from prompted input phrases, which I learned in my technical travels. However, I'm a software developer and not a scientist, so this section may contain inaccuracies.

The basic building block in transformer AI models is called an embedding, which is a fixed-size vector of floating point numbers representing the item associated with this embedding, such as an input token or the entire sentence. Each element of an embedding captures some aspect of the associated item, such as whether it is a noun, an animal, can fly, and so on, but without these traits being explicitly identified as such. In other words, embeddings for "cat" and "dog" will have similar values in certain elements that capture their shared trait of being animals and will have a similar relation to an embedding for "pet", for example.

Transformer AI models generate responses in two distinct phases. First, an encoder converts a text prompt into a sequence of embeddings that captures the meaning of the phrase from the prompt. Second, the decoder iteratively generates a response, one output token at a time, maintaining various contexts between iterations to arrive at the most relevant next word for the current context.

The correlation between questions and answers is reflected in the weights and biases of model nodes used by the decoder, allowing the model to generate relevant responses for various prompts. In contrast, the part of the model used by the encoder is more generic and is trained on diverse datasets that include a variety of linguistic patterns, contextual meanings and relationships between them, which works well for extracting token features.

Encoder

The encoder generates a sequence of contextual embeddings from the input phrase, capturing the multifaceted semantic meanings of that phrase, taking the following steps during this process:

  1. An input phrase from the prompt is broken down onto tokens. Each token may be a word, part of a word or even punctuation.
  2. An embedding is created for each of the input tokens.
  3. Each token embedding is passed through the model's layers in parallel.
  4. Each hidden layer of the model contributes to the embedding in describing token features, which is achieved by combining the current value of the corresponding embedding element, its positional information, the shared attention state, and each node's output.
  5. All token embeddings that were processed in parallel are assembled into a sequence of contextual embeddings (a sequence of hidden states).

These sequences of contextual embeddings may be used on their own for grouping semantically different documents, semantic searches, sentiment analysis, AI assistants, and other similar uses that require semantically similar phrases, such as "I paid my phone bill" and "I transferred $50 to Bell", to yield similar responses.

Newer models will also enhance the resulting sequence of contextual embeddings via chain-of-thought mechanisms to include sub-task information into the embeddings or insert task embeddings into the sequence, which will work for phrases like "a train travels distance A in time B; find how long it takes the same train to travel distance C", which describes a two-step process in finding an answer.

Decoder

The decoder generates human-readable phrases from sequences of contextual embeddings and the current conversation context.

The decoder takes the following steps during this process. A general decoder state is maintained to carry the original sequence of embeddings, the output token sequence that is being generated and some other data. This state is distinctly different from the decoder hidden state, which is an embedding being cycled through the decoder layers.

  1. The attention mechanism in the decoder computes weights for each contextual embedding in the sequence of contextual embeddings generated by the encoder, using the decoder output embedding in the process. A special initial embedding is used for the first iteration.
  2. The attention mechanism combines these weights into a single embedding-sized context vector. This embedding represents the decoder hidden state.
  3. The decoder's input layer receives the current hidden state, and all hidden layers are traversed, guiding the process towards the most likely output node in the output layer.
  4. The output layer has as many nodes as there are possible output tokens (model vocabulary), and each of those output nodes has an associated output embedding that describes traits of that output token.
  5. The output token is added to the list of tokens in the decoder state.
  6. The process repeats with the new output embedding, until the entire human-readable phrase is generated, which is determined by end-of-the phrase output token.

The decoder attention mechanism is different from the self-attention mechanisms in the encoder and the decoder, which operate across hidden layers.

The decoding process is sequential and it generates one token at a time. Machine Learning reasoning mechanisms are applied during this process to improve the contextual relevance of the response, working together with various linguistic algorithms to ensure well phrased responses.

Multiple possible sequences of output tokens may be generated as possible alternatives, so they may be evaluated together against one another (e.g. beam search) to select the most relevant sequence of tokens.

The final response phrase is post-processed to further improve the grammar and tone of the final response.

Good food - good AI

While the encoder and the decoder are so incredibly well designed and implemented that they look like magic at times, the overall quality of conversations will, at the end of the day, still be defined by how well prompts are matched against responses, which is captured in the distribution of weights and biases in the decoder's layers, so poor training data will produce misleading or incomplete answers regardless of anything else.

Herein lies the problem with questions and answers for many software development topics, which are often obtained from sites like Stack Overflow, where a person asking the question will mark responses as answers, but because they are not experts, many of those answers can be classified as quick fixes rather than best practices, but these answers still ends up in the training data.

The quality of responses in any AI model is only as good as the data it was trained on. In other words, if an AI model were trained only using manuscripts from 1500s, the AI using that model would say just as confidently that Earth is resting on a giant tortoise floating in a body of water.

On top of that, sometimes corporate overlords decide that they can nudge science to work faster, like what's described in this TechCrunch article about Google urging contractors evaluating quality of AI responses to do so when they don't have the expertise in some subjects.

https://techcrunch.com/2024/12/18/exclusive-googles-gemini-is-forcing-contractors-to-rate-ai-responses-outside-their-expertise/

Web searches masqueraded as AI responses

Not all AI responses are created equal and some of them are not even derived from carefully curated training data, but instead are just summaries of top-ranked results in web searches. These responses may be identified by the presence of footnote links in AI responses.

These types of responses will be a hit-and-a-miss in terms of quality, and, in general, will be worse than if one would actually search the web, because in this case most people would assume questionable nature of web content and would look for additional quality indicators, such as how the content is structured, are there any corroborating comments in those posts, links to respectable sources, level of detail, and so on.

When such answers are served as AI responses, people tend to assume that those responses were verified via the magic of AI and will more likely accept them as-is, whether they actually make sense or not. Here is an example of such response, in which I asked Copilot where Java virtual threads are preempted - before the actual I/O call is made or in the process of making a call.

Where Java Virtual Threads are Preempted
Where Java Virtual Threads are Preempted

The link in the response was as follows:

https://blogs.oracle.com/javamagazine/post/going-inside-javas-project-loom-and-virtual-threads

, and the only relevant text on this page about how blocking I/O calls are handled is in this paragraph, which provides no detail about sequencing where a virtual thread is preempted.

Instead, virtual threads automatically give up (or yield) their carrier thread when a blocking call (such as I/O) is made. This is handled by the library and runtime and is not under the explicit control of the programmer.

I followed up with Copilot, trying to indicate that this page provides no conclusive description about how virtual threads are preempted, and Copilot solved this problem very trivially - it just removed the link and generated the same response, but without a link footnote.

After searching the web some more, I found the article below, which goes into a great detail of how Java virtual threads are being handled, along with showing stack traces, which indicated that blocking calls are converted to non-blocking calls, and the initial I/O call is made in the context of the virtual thread, after which the virtual thread is preempted, with the completion callback set up to run in the context of the platform thread. When the I/O completion callback is invoked, the platform thread looks up the matching virtual thread by a file descriptor, and restores the call stack of the virtual thread, so it can return from what the calling code will perceive as a blocking I/O call.

https://inside.java/2021/05/10/networking-io-with-virtual-threads/

When I pointed out this logic to Copilot, it immediately confirmed that this is how it works indeed and included the above link into the response footnotes.

Responses based on Q&A discussion forums

The next level of response quality comes from various Q&A discussion forums, such as Stack Overflow, being used as training materials. People ask questions in these forums and accept responses if they work for their specific case, which is not necessarily what one would consider as best practices.

Here is an example of a question about including application text output safely within the generated HTML, which is a topic many developers struggle with, perhaps because of many nuances in which text is embedded in which context.

For example, if one wanted to output a string A<">B in HTML, it would have to be encoded as A&lt;&quot;&gt;B, but the same string used in an inline event handler function call would have to be encoded for JavaScript first, and then for HTML, like this onclick="fn(&quot;A&lt;\&quot;&gt;B&quot;)".

Note the backslash before the &quot; character entity in the middle - this is an example of that multi-context encoding. In this case, the browser will apply HTML decoding first, which will yield fn("A<\">B"), which the browser will then pass to the JavaScript engine, which in turn will apply JavaScript decoding to construct the parameter string with the value A<">B.

One can see the complexity of the process from this example, which does not even include all cases, such as emitting text within a block of JavaScript code or within JSON blocks, and other similar situations. Here is what Copilot responded when asked how to do it safely.

Safely Output Application text in HTML

This is a terrible answer, with a bunch of omissions and some misleading advice thrown into the pile. The first point is correct, but it fails to mention that in addition to the HTML encoding, other types of encoding may need to be applied, as described above, or, alternatively, data attributes should be used instead.

That is, instead of dealing with all of the types of encoding, one will greatly simplify their development effort if only HTML encoding is used to store values in data attributes, and then use JavaScript to obtain dataset values. This will not work for large values, but in most cases this approach is all one needs, so the first item is correct, but incomplete.

The response goes downhill from that point on. Using text nodes is a moot point for data coming from the server because this data is supposed to be HTML and/or JavaScript encoded as needed by the server, so no special magic is required when operating on this data at the element level in DOM, such as creating text nodes. For data received from the client, it must be validated through the server before anything can be done with it, so trying to juggle properties like innerText, etc. before server-validating data is a good recipe for a future vulnerability.

The 3rd item is also not good - any application that takes bad input and removes some of the dangerous content is failing to validate that input properly. That is, if bad input is detected, it should be rejected as a client error and not just sanitized. Moreover, using the DOMpurify package is a bad idea because it operates at the DOM level, so unvalidated input will be parsed via some other package, such as jsdom, and only then traversed in memory by DOMpurify, which gives a lot of attention to the potentially adversarial markup. User input should be validated and not sanitized. A better approach is to use a stream parser that will indicate if any of the bad stuff is found, which can be used for rejection without having to build any parse tree structures.

The 4th item is titled correctly as server-side validation, but then it slides back into input sanitization. Server-side validation should focus only on validation and should fail any input that is not allowed. It is also Ok for such validation to receive safe HTML if this is what the application expects, such as basic formatting attributes in user comments, or even unsafe HTML, which simply should be encoded, such as in user comments on a vulnerability sharing site, which should allow user comments like "The attackers submitted <script>...</script>, which was caught by X", which should not trigger any validation errors because any dangerous code is safely HTML encoded and will appear just as text.

Lastly, CSP (Content Security Policy) does not have anything to do with generating safe output in HTML and should be included in such response as an additional measure that may help mitigating XSS vulnerabilities in a poorly maintained application, or in a well-maintained application served over unencrypted channels.

Complex topics will usually present this kind of challenges, but for more straightforward ones responses will be quite good and helpful, such as asking how to loop in Bash or in Windows, or how to print values to the console in C++, although the latter will still favor std::cout instead of std::print, even when explicitly asked for C++23, so some follow-up would typically be required to get better answers.

Responses based on human-reviewed content

Topics for which AI models were trained on high-quality human-reviewed content is where AI really shines. If one wants to learn about human biology, machine learning, fundamentals of software design, and other topics covered by training materials that do not directly come from Q&A discussion forums, AI will provide a wealth of information to learn from.

For example, if one wants to learn how neurons work in a human brain and whether their binary nature presents any challenges in how humans process information (compared to AI model layer node outputs), and this knowledge is just intended to further one's understanding of the world - this will be the most illuminating conversation that was not possible just a few years ago, because all this knowledge existed only in specialized literature and was not covered in good detail in layman publications.

However, there is a catch - all AI responses will look like they are coming from an expert, but they will range from spot-on answers, to sort-of relevant, but not the best-approach answers, to very questionable answers that are better to stay away from, and there is no way to know which category each answer falls into if you just are trying to learn a new subject.

Those who have strong understanding of causal relationships between entities, events and outcomes will benefit the most from conversations with AI because they can recognize responses do not add up in some way, and will ask questions to either steer the conversation into a different context, or, perhaps, to learn that their current understanding that raised this red flag is not quite correct. I find these follow-up questions invaluable in crossing the t's and dotting the i's in these conversations.

Those who mistake AI for a human expert and apply provided answers as-is in their field, will probably not do very well when their work is examined by actual human experts, be that university professors grading papers or team architects reviewing submitted designs and code changes.

For example, if one asks AI how to structure a logger service, the response will be quite good and will describe the performance implications of such service, the need to asynchronous logging and will provide good advice on how to align log records by their original time stamps, and other relevant details. This is likely because various existing logger packages are quite similar and well implemented for most uses, so AI just captures this coherent view on loggers in its responses.

If one asks AI about more diverse areas of software development, such as how to implement a database access layer in a server application, AI will provide frequently described design patterns in a response that will sound like a solution, rather a brainstorming session to choose the best approach for the given requirements, which is what one would really want from such conversation.

For example, a typical response for this question will outline the data access layer, repository layer, service layer and the application layer, but will use generic terms in describing how these layers interact, such as that the repository layer provides "higher-level abstraction than the data access layer", and provided examples will not show good layer separation details. It is not a bad advice, even though it is not as clear as it could be for various types of applications, and examples often will muddy the waters by showing fully assembled entities, like a purchase order, being used in the data access layer.

A good human architect would describe the repository layer as a layer responsible for assembling entities from their records, and would highlight that for applications with a dozen entities one does not even need a repository layer because entity service layers can perform these operations, which keeps implementation simpler, without introducing any strong coupling between layers, while applications with numerous entities will definitely need the structure of these entities maintained in their own entity readers/writers, which would be akin to a repository layer.

It gets really bad for topics that are commonly poorly described in various online discussions and do not have much high-quality content on the web to augment those discussions. This is where AI will give a really terrible advice, like "releasing a package v1.0 from a branch release/1.0" or having "release/1.0 and support/1.0" branches, and other bits that cannot be described other than highly incompetent advice.

Coping mechanisms

If you see footnote links in AI responses, you might as well just do a web search to obtain better answers. Worth noting that adding "No web search. ..." to the prompt often results in the same web response, but without footnote links.

Over-specifying questions will quite often result in more specific answers, such as adding "Keep your responses short. How does ..." or "Focus on the data access layer. What is ...".

If you are not sure in some response being good or not, being forceful in pointing out the discrepancies in a response may push AI to respond with what you want to hear, which is not necessarily helpful in what you are trying to learn. That is, if some response doesn't sound quite right, asking in the form of a question may provide a better path at finding the true meaning, such as "Is it fair to say that X does Y?".

Sometimes AI gets stuck on repeating the same answer that you know to be incorrect or irrelevant, no matter how you rephrase the question. A good way out of it is to refocus the question on a specific detail of the response. For example, if you ask how AI adds numbers, it will likely to go on about node outputs and activation functions, but without providing concrete details. Asking specifically whether adding 10 and 15 maintains number 10, 15 and 25 anywhere in memory, or whether the model merely infers characters 2 and 5 given character strings 10, 15 and add, will nudge this conversation towards a more clear answer.

From time to time bad answers will keep coming on a loop, regardless how elaborately the question is rephrased. This is a good indicator that the AI does not have the answer, but given that AI does not understand rejected premises at their core meaning, it will keep generating slightly varying incorrect or irrelevant answers for as long as you are willing to type your arguments. Take this as a cue to start looking for alternative ways to answer your question.

A partner or a competition?

AI does extremely well in many areas, such as identifying patterns, summarizing documents, finding specks of information in volumes of data, and many other areas, but the trick for getting the most of it is in deploying AI systems in a way that improves efficiency and productivity, which seems to present challenges in existing observable deployments.

For example, many cable, phone and energy providers offer their support via call and chat centers, which are notoriously bad at their job beyond suggesting to reboot the equipment or connect some cable. Replacing call centers with an AI system sounds like a very welcome move for most consumers, but with a couple of caveats.

It is expected that AI will not be able to solve some of the more difficult problems, so if it is deployed in such a way that it will guide customers towards simple solutions, but would recognize problems beyond its purview and forward these calls and chats to human experts who can provide more nuanced troubleshooting, those AI systems would be a gift to all of us who suffered enough through hours of mind-bending incompetent conversations with the first-level support personnel.

Unfortunately, many corporate minds will realize that they can save money on not having to hire those few experts and instead keep on a few of the first-level support people at a much lower rate, and it seems that many go for that because it looks good for the bottom line. Between closing support cases and having to solve messy problems, the former is more likely to win in board meetings.

Hopefully, corporate decision makes will realize sooner than later that AI works best when it is paired with human experts in their field, and not when experts compete against AI.

Final thoughts

AI is a fantastic technology that allows people to learn very complex subjects via interactive Q&A sessions, without having to sieve through specialized studies and research papers, which works out incredibly well for self-education in a broad range of topics.

However, it is somewhat paradoxical that those who already have much knowledge in a variety of areas will benefit the most in these conversation because they can navigate efficiently between incomplete, incorrect and good responses, while those who consider AI as an equivalent of a human expert will likely end up applying some of the bad advice they got from AI in their work or at school.

Even a good human expert is, well, a human, and they can make mistakes, but this is where there is a difference - if one points out to a reasonably-minded human that they have evidence that contradicts what they just said, that person will review their logic and the evidence, and will either point out a flaw in the evidence or will reconsider what they were saying - that is how human intelligence works.

AI, on the other hand, in the same situation will keep repeating the same points in different ways, sometimes AI may even appear to agree with you within the current conversation context, but when a new session is started, it will provide the same responses as before. Nothing will change semantically in those initial responses until a new model is trained or AI inference code is updated.

Maybe one day AI systems will become sophisticated enough for a Skynet type uprising, but today AI merely recombines in the desired tone the bits and pieces of human-created content it was trained on, and the true danger of AI is not in AI "becoming self-aware", but in humans applying AI in ways that will be harmful for humanity, be that the incompetence in applying bad AI advice, or deploying AI systems in a way that will make customers suffer the brunt of misleading AI responses, or the explicit use of AI to generate fraudulent content.

Humans grow and improve intelligence throughout our lives, so we apply any specialized knowledge acquired in schools, at work or on the streets within the framework of our overall intelligence, including the common sense and ethical "subsystems", which contribute to creating multifaceted solutions and allow us to extend these solutions beyond just the acquired knowledge. This is how ancient humans were able to look at shadows and create a new knowledge point that the earth is round.

AI does not have that additional growing-up experience and must rely solely on training data to acquire knowledge, which is captured in millions of weights and biases in modern AI systems and actually encodes the human intelligence extracted from the training data. This enormous database of human knowledge allows AI systems to apply the human intelligence in ways that may look like creativity, but they really have as much creativity of their own as a human would exhibit while working on a jigsaw puzzle of a beautiful masterpiece painting.

Comments: