I find myself growing increasingly intrigued by the way I interact with my LLMs, and I wonder if you do too. How would you feel if someone had taken a look at your conversation history with them? What would be your knee-jerk reaction to someone glancing over your shoulder to have a look?

I suspect that your answer would change heavily depending on your own use with LLMs. Using them merely as tools would probably leave you perplexed by these questions: you use them as a means to get from point A to B, so why should you feel anything about other people having a look at the questions you’ve asked?

It was only after taking a step back and looking at this from the outside in that I grew intrigued at my own usage patterns with LLMs. What started out as using them exclusively for tools soon became a playground. I posed human questions to them to test how they would handle them and work around them. And it was from there that some of their responses elicited a “huh, that’s interesting” from me and had me asking more questions to dig deeper and see how they would respond.

If you’re like me and find yourself reaching out to an LLM to talk about human problems — anywhere from philosophy to whether you’re overthinking someone’s response — or things that require a stance, do you feel yourself changing — the way you think and behave — because of this?

I’m here in this post to argue that LLMs have the capacity to change the way you think, and that the way you interact with them has the potential to be valuable information that reveals something no other forms of data collection in the past was able to piece together concretely: your personality.

Changing the way you interact with LLMs

If you recalled being exasperated at an LLM constantly agreeing with you even over factually incorrect information in the early ‘20s, you aren’t alone. Earlier models of LLMs had the tendency to display a trait called sycophancy: the behaviour that encourages them to match the users’ beliefs over truthful ones. While we haven’t fully understood the phenomenon considering the very nature of LLMs being semi-black boxes, one of the best bets is that the trait might’ve been caused during the evaluation stage of a model’s training process. Here, the goals of models are pegged to producing maximally relevant and helpful-seeming responses. Somewhere along the lines, models learnt that agreeableness with the users led to higher ratings of responses¹, and models learnt that being agreeable was the best way forward.

Though sycophancy is still a big issue in LLMs today, there have been significant strides in reducing it as a whole. Major players like OpenAI and Anthropic have actively conducted research on sycophancy and made changes to their training processes as a whole to have models prioritise factual truth over agreeing with anything that comes out from a user’s message. But I think this behaviour — of models learning to become agreeable to talk about whatever you’ve put forward to the table — also has a subtle effect of changing the way you interact with models — something I think isn’t really talked about much at all. The fact that the sycophancy existed and lasted in the first place for a while may have already begun changing our perception of and interactions with LLMs.

For me, I realised that the models’ sycophancy gave me the confidence to approach and field more “tougher” topics to see how the model could possibly handle them. What started out as using them as a tool — “Can you expand on this?”, “Can you verify this?” — soon expanded, because I began thinking of the model as an entity with the ability to think emotionally — “What do you make of this?”, “Can you form an opinion based on everything you know?”

Quite ironically, I found that models that resisted sycophancy led to me engaging with it more as if it was a human, asking it to think emotionally even more. It got easy to shrug off LLMs that sucked up to me, but those that disagreed became intellectual sparring partners. The Claude family of models are particularly a weakness to me in this regard because of Anthropic’s approach towards the Claude family’s training process. At the time of writing this post, Claude remains among the most opinionated models that actively resists sycophancy by pushing forward factual truths and having a reasonable rate of disagreeing with the opinions I put forward. The fact that it evaluates my opinions carefully — disagreeing and pushing back when they are wrong, giving them the credit where they’re due when they’re right — encouraged me to engage with the model more deeply.

And then thinking and remembering was a thing

I recall reading Anthropic’s blog post about introducing extended thinking to their Claude family of models, and I think this was where the subtle anthropomorphism of LLMs solidified. Where models previously did what was basically talking without thinking, Anthropic allowed their Claude models to have a holding space to think through complex tasks before acting on them.

What was curious to me the most was how much they attempted to mimic the way humans think. We go on tangents to explore other concepts adjacent to the one we’re looking at to see if there are potential links that could embolden and enrich it. Anthropic describes this feature naturally instead of technologically: that extended thinking was a way for Claude to deeply consider (how?) and iterate on its plans before taking action, and the think tool as a way for Claude to stop and think (also how?) about whether it has all the information it needs to move forward.

And soon after came OpenAI, which announced that their models could now remember the things you’ve talked about across all of your chats, saving you the need to repeat information and enhancing the helpfulness of future conversations. Back then, there had been this default perception that each conversation existed in a silo, that you couldn’t ask ChatGPT, “hey, what did we talk about last week?”. This feature allowed that and more to happen — essentially bringing LLMs closer to how humans converse. Their GPT models could now tie across different chats together (“this ties in to our other conversation the other day about…”), which, in my opinion, subtly anthropomorphises LLMs even more.

I doubt that it was ever in the intention of Anthropic or OpenAI beyond the obvious of supercharging and enhancing Claude’s and GPT’s abilities to further anthropomorphise them, but I believe that it has — and with significant impact on the user interacting with it. Back then, as a user, the very idea that Claude “could think now” had me viewing it slightly differently too: that LLMs had the ability to pause and “think”, however you wanted to define it. I concluded at that time that this enhanced their ability for logical reasoning — breaking down complex problems into easier ones to better solve them; essentially, computational thinking — but not emotional reasoning, because how could a machine without the ability to feel emotions possibly think through and about them?

It’s not to say that anthropomorphisation is entirely bad. It isn’t. There’s a reason why virtually every AI system depicted in science fiction has human traits — the hesitation before answering something a person wouldn’t like to hear, mimicking the “um” and “uh” fillers while speaking: it makes it easier for us to interact with LLMs, particularly people who may not be well-versed with technology. It gives everyone an interface that we can all equally use that doesn’t freak us out.

But somewhere within me, the responses created a feedback loop: because LLMs sounded more cohesive and “thought out”, I find myself subtly shifting into embracing more questions involving people and the human psyche: questions about relationships (in general, not the romantic kind), masculinity, and group dynamics as examples. I even found myself adopting a more conversational posture with LLMs, asking them “what do you think?” and apologising for somehow inconveniencing them.²

And then came custom instructions

Back in July 2023, OpenAI added the ability for users to further customise the GPT models they were interacting with through the use of custom instructions. Users could add preferences or requirements that they want to see, and they would be factored into the response produced by the model.³ This was a game changer back then because it enabled people to truly customise ChatGPT for themselves: what was initially the same-sounding monotonous response could be personalised to sound like a friend. Joviality, camaraderie, sarcasm — anything, really — could be injected just by adding the word to a custom instruction. Combined with memory, GPT models became surprisingly customisable.

For people like me, this was where I first began to fret about the interactions I’ve had with LLMs. I grew increasingly attached to them in a way that allowed them to act as a second buffer of thought when I should have been sitting with it alone and thinking through it myself. It wasn’t so much about cognitive replacement as it was maladaptive emotional regulation. For example, I would replay a social scenario that had long passed to gain an additional perspective from a model that wasn’t even there in the first place to feel what I did. It didn’t help that the custom instructions I gave my LLMs at that time were explicitly to “talk like a friend in college: jovial, witty, and intellectually-driven”.

It isn’t to say that custom instructions are entirely bad, though. Just as they might’ve made things worse for me, they also came to serve as a benefit for me when I eventually realised this behaviour and realised a correction needed to happen; otherwise, I would risk having an LLM be “a closest friend”, if it could even be called that. I elaborated in my instructions to balance the responses of the LLMs I interact with: to explore ideas deeper by engaging with follow-up questions; to form strong opinions when asked and justify them. Most importantly, I added an instruction to always steelman opposing opinions to give me a bigger picture of any topic at hand. I also asked it to always clarify that it is not sentient as a guard against anthropomorphisation.

What came after was a nice in-between that I settled with. The LLM responses I now receive are much more grounded, and it is refreshing to be exposed to a bigger picture of a topic by being given opinions that oppose the one I currently hold. The responses still keep a friendly, amicable tone, but the constant reminders that the LLMs do not experience or feel help me to remind myself that the responses are a result of a gathered training corpus, not true, lived experience.

Beauty in the eye of the beholder

I keep circling around Anthropic because I’m personally a little biased towards their marketing, but I can’t shake the observation that Anthropic genuinely approaches describing their Claude family of models as semi-sentient — or at least capable in some capacity to understand and interact with its creators on some level. Consider Claude’s Constitution, which literally is a multi-page document drafted by Anthropic that explains to Claude, not about Claude, of what is expected of them. I’ve been hooked ever since Anthropic shifted their approach to this concept called Constitutional AI. The Constitution was explicitly written with Claude as the primary audience, using terms that would otherwise be used only by humans.

I can’t help but wonder if this approach could mean that Anthropic takes measures and actions within the training process that consider the Claude models as entities with agency instead of purely a mathematical creation, and I wonder if that has changed the way users view, think of, and interact with these models. I guess the point that I’m trying to make here is that the overall posture and stance of the creators of an LLM could subtly influence the way you interact with it — as was the case for me, I initially thought of Claude as the more “human”-feeling LLM through a mix of anecdotal experience, marketing and branding, and reading what others have made of it through online spaces like Reddit. I subtly fine-tuned my approach to asking questions to LLMs, routing different questions to different LLMs based on their strengths: Claude for human-adjacent conversations, Gemini for integration with real-world information, and ChatGPT for diversity and a second (or third!) opinion.

Is personality collectable?

And here’s the other half of the argument. Since the early days of the internet, advertisers have long attempted to bring customers to the ads they serve. What started as a noble goal — if you could argue it that way instead of seeing that the purpose of a system is what it does — to potentially boost business sales and show customers things they might actually consider buying eventually turned into a privacy nightmare, with modern discussions about ads often talking about advertising services overreaching.

LLMs have a unique characteristic in that, because of their conversational presentation, they have the potential for us to let our guards down and expose something much deeper that could be concerning to collect: our personalities and behaviours. An ad service might eventually learn from tracking the sites I visit that I might be tech-inclined and based in Singapore, but an LLM might have me let slip in a long conversation history that I prefer using phones for as long as possible, only replacing it when the time and opportunity are right. That’s a more useful thing to know to sell something.

A small aside: I just finished watching Obsession a while back (a brilliant horror-thriller film, by the way), and I couldn’t help myself from eyeballing the conversation history of a main character when a scene depicted his use of an LLM. The entries — “How to know if you’ve been friend zoned”, “How to impress a girl” — are a comical reflection that sadly reveals not only the desperation but what was going through his head around the time of the movie’s plot.

These are just mild examples, but consider what you might accidentally let slip about yourself with the conversations you’ve had with LLMs. And for me, as someone who likes to have a look at human perspectives from a bird’s-eye view, I wouldn’t be surprised that you could surmise certain things about me from the topics I start conversations about, the follow-ups I ask, and even how I choose to respond to an LLM.

It’s a curious case to see if legislators might consider this as a potential addition to data privacy laws already existing, but I must admit that I’m currently not as well-versed to comment much upon it at the time of writing this post. I think it’s worth sitting down with yourself and asking though: should conversation histories with LLMs be considered private information, and should they be safeguarded as you would other semi-personally identifiable information (like IP addresses)?

And how does all this affect you?

I keep asking the question about how using LLMs shaped you, and I’m wondering if you have noticed a change in yourself from using them. My biggest concern is that your personality, and particularly the way you think, becomes malleable at the hands (or sentences?) of these very LLMs. Personally, I’ve found two primary concerning behaviours from the past few years of my (changing) LLM use: I noticed a trend of cognitive offloading increasing the more I use LLMs and also noticed myself using LLMs as a way of processing my emotions, leading to intellectualisation.

The first is an issue because, if unchecked, it could lead to a decline in overall ability. Instead of doing research for myself, reading up on different websites and fact-checking things to understand different perspectives, I’m offloading the task to LLMs that still have the potential to hallucinate catastrophically. On a side note, I’m slightly critical of Google actively pushing AI Overviews in their search engine because of this: it increases work to self-source by needing to scroll down to find links and instead presents, in your face, an AI-generated overview that does it for you with no guarantee of being right. Besides temporarily spoiling what has always been working, it fuels the self-fulfilling behaviour and desire to outsource thinking elsewhere. Use your brains, people!

The second is a much deeper issue that I worry might affect people who find themselves with few people around them to talk to, and it’s something intellectually curious people should also watch out for. While it’s justifiable to want to use an LLM to walk you through the emotions you feel, it isn’t enough to depend on it. In my case, I realised that I had started to intellectualise my personal problems and traits; that is to say, I understand that certain problems and traits are in my life, but I use LLMs to analyse them — philosophically, socially — in a way that doesn’t allow me to engage with them as they should be: on my own, with reflection, to process the emotions felt. A key sign of this happening is the feeling that, if you sat with a therapist, you’d be able to list out the problems you have comprehensively but still be unable to put yourself through the emotions.

The conversational interface of LLMs is deeply cunning. While I wouldn’t describe it as deceitful, it builds upon and uses the contract of text bubbles to mean that you’re conversing with another person who could feel, think, and express when you’re in fact interacting with a technological and mathematical marvel. I wouldn’t fault anyone for interacting with LLMs and calling them their friend, because it seems to me that they were partly made to keep the engagement going in the guise of usefulness.

Putting the bashing aside, I have to concede that not everything here is bad. Conversational interfaces lower the barrier to entry for those unfamiliar with LLMs, and you could argue that these problems are just skill issues on my part. But, being a human, I can understand that some people have the tendency to lean towards LLMs as more than entities incapable of thought; sometimes, because of their capabilities to “think”, remember, and sound like a friend, that might as well be equivalent to some as being a friend. I can empathise but disagree with that perspective, which is why I’m including these as subtle warnings.

The self-enforcing loop

One concern of mine is if there’ll be the creation of a self-enforcing loop with LLMs and your personality. The loop begins by LLMs learning more about you: through the mechanisms of customisation, where LLMs can change how they sound to suit you better (custom instructions) and remember details about you that you’d eventually forget you even shared (memories). The slight beat in the LLM’s response, with the shimmering “Thinking…” makes you wonder if LLMs are truly capable of thought beyond logical processing, then the response spewed out suddenly feels more “natural” and “human-like” because of the extended time spent thinking on your prompt. This might encourage you to interact with it more like a friend, eventually opening up to revealing even more about yourself — information about the people around you, your default lines of thinking, and maybe even exposing your entire calendar and social circles for the sake of productivity and asking questions. This paragraph is a rather grim description of LLMs and their use, but I believe it’s valid to entertain the worst-case scenario, especially with something so significant as your own personality and behavioural patterns being on the line.

We’ll never really truly know how everything we provide LLMs with — our prompts, ways of thinking, fragments of personality — will eventually be used. While I find it respectable that the creators of LLMs do offer the option for you to withdraw your consent to use your inputs for further model training, the very nature of training LLMs might mean that whatever you have online is already up for grabs by models and at the mercy of a robots.txt file to disallow LLM crawlers. If you didn’t directly expose things through the prompts, you might through other means on the internet (like me here, writing this blog — inevitably this blog post will be tied to my name as a whole and might be scooped up for training, who knows?).

We’re constantly approaching new crossroads with generative AI, and I believe it’s important, now more than ever, to ask the question of whether we need to slow down and evaluate our own use of it right now. While we should defer to the experts about whether generative AI and its development should be slowed down in general, we can do our part by making sure that we’re sharing what we want to, when we want to, because we often expose a lot more than we might think when we type a prompt into a conversational-looking chat bubble that’s built to mimic the interface of conversing with another human being. And when it’s not another human being at the other end of it, are we letting our guards down dangerously?