In recent years, large language models based on the Transformer architecture were trained on large corpora of text from the internet. Despite being trained for the simple task of predicting the next word, these models show astonishing capabilities. They can generate long coherent text and solve tasks given only a description and possibly a few examples. I will describe the architecture, training process and results that come from these models which lead many people to ask: are they indeed capable of a form of understanding, or are they just stochastic parrots? I will then argue that coding tasks provide a good testbed for this question. I will show how large Transformers have been adapted to code and demonstrate both their successes and failures. I will then discuss a few recently introduced techniques that significantly improve coding and reasoning performance of large language models.