I have started playing with Chat-GPT. Who hasn't. I made good use of it to practice my French, and I completely misused it for a coding problem. Here was my experience.

Language practice

My first use case benefits from its strengths, and is not harmed at all by its weaknesses. I have been trying to learn French, and the ability of chat-gpt to simulate conversations and tell stories seems like a great way to practice. And the fact that it can get facts wrong doesn't affect me at all. Facts were not at all relevant to the story it told me about Ahsoka Tano joining the Three Musketeers.

In my first time using the site, I typed "Bonjour" to see what would happened. It greeted me in French. I was kind of at a loss for what to say next, in my limited French, hence the need to practice. Eventually I came up with "I like exercise", it said exercise was good, and asked what kind of exercise I liked, and we talked about running a little bit. I then asked if it liked coffee. After telling me that it was incapable of liking anything, it talked about some things other people like about coffee and we talked about that for a bit.

In another session, I asked for stories in French. I used a hybrid approach here. I wanted my side of the conversation to be in French, but I didn't have all the words I needed, so I used Google Translate, which gave me this key phrase "en utilisant des mots que les enfants peuvent comprendre" - use words that a child could understand.

The next thing I want to try is scenarios: Pretend you run a coffee shop, what would you say if I were a customer. I tried this in English, to see how it worked. I was surprised that it seemed to take a little longer to generate each response than it had in other conversations that it had. Telling it I wanted a medium dark roast and a quiche made it think for longer than it did when I had it generating complicated code examples. The responses were a lot more wordy than I would expect in a real-world situation, but that is probably ok for my use case.

Code generation - my bad

When generating code, correctness becomes a priority, but I don't think perfection is essential. I never get anything right the first time, and I don't need an AI assistant to either. Anything that helps me converge to the right solution quickly is beneficial.

I don't know if my first use of code generation was a net benefit. I did eventually get the solution to work, but it took a lot more time and effort than I expected. My workflow in this case was terrible, but improving my process and improving my prompts, I expect I will get great results.

I was trying to send an mp3, either as a blob or a file, from a react app to a python lambda. I am not a javascript developer, and the combination of binary data, API Gateway, AWS Lambda, and Python was too many moving pieces that I didn't know enough about.

After spending hours googling and experimenting with no real progress, I decided to stop for the day. Out of curiosity I decided to see if chat-gpt could come up with a solution. I asked for a python3 lambda function that accepts an mp3 file and saves it to s3.

The code it gave me looked funny, and I couldn't figure out why. After a little discussion I realized that what it was doing was responding to a file being uploaded to s3, and copying that file to a new location.

Before I figured out the problem, I asked it to generate javascript to invoke the function. From that point on, every request that prompted changes to the python code, the javascript code was also updated.

Eventually, I gave it more detail about what I was trying to do - that I wanted to send an mp3 file to Amazon Transcribe. It generated the lambda function, the javascript code to call it, and the yaml specifications for API Gateway.

I said "wow, this is amazing," but I was tired and decided leave it till the next day to incorporate the changes into my code. The next day, after modifying my existing code to use what chat-gpt had generated for me, I was not getting any binary data in my lambda. At the micro level, this may have been my own fault, because I didn't understand the changes it made to API Gateway, and I may have misapplied them. At the macro level, it was certainly my fault, I was using chat-gpt completely wrong.

What I learned

Generating the code ahead of time, and then trying to tailor it to an existing project is completely the wrong approach. What I should have done was type (or paste) the changes into the editor as it was giving them to me. And if I encountered problems, I would have been able to get feedback on them immediately. As it was, I was able to get improvements to the solution when I asked it questions and requested changes.

One thing I discovered, but I don't think I could put it in words before today, is that search is general. With Google, I inevitably find pieces of the solution I am looking for, and I have to extract what is relevant from several articles, and then synthesize them into a solution.

LLM is specific. The more details I gave about the problem I was trying to solve, the more help it can give me. I expect that if I had pasted in my yaml template and asked it to modify it to accept binary data, it would have. Or, rather than having it modify my existing code, I could have used its solution as a stand alone solution, ironed out any kinks, and then used that to inform the changes to my existing project.

And if you care, I did finally get my code working. I base64 encoded the audio so I could pass it as a string, and then decoded it in my lambda function. The process of getting that working reinforced how poorly search works right now. Many of the "solutions" I found were deprecated, and didn't work anyway. It took several iterations of search, try, adjust, search before I had working code.