One night in 1999, Jim Broadbent and I were in the audience at the New York film festival. Our Gilbert and Sullivan film, Topsy-Turvy, in which Jim plays Gilbert, had been screened the previous evening. Tonight it was the turn of Being John Malkovich, Spike Jonze and Charlie Kaufman's delightful surreal comedy about people popping through a magic door and finding themselves inside John Malkovich's head.
Suddenly, Jim and I looked at each other. This was a quintessentially Gilbertian "magic lozenge" plot. Having immersed ourselves for the previous couple of years in the life, work and preoccupations of WS Gilbert, we were well versed in his predilection for devices that turn you, with dramatic consequences, into somebody else, or into a different version of yourself.
The magic lozenge was the bugbear of Gilbert and Sullivan's working relationship. Every time Gilbert proposed the idea, Sullivan rejected it. He disliked fantastical devices that tampered with reality. He always wanted "to set a story of human interest and probability", as he wrote to Gilbert in 1884.
Of course, they had done the lozenge in their first full-fledged collaboration, The Sorcerer, with its love philtre. But what Sullivan failed to spot was that every one of the subsequent operas was a lozenge story without the lozenge, in that they all involve transformation of identity, and that, far from failing to deal with human emotions, Gilbert's metaphorical world enabled him to do precisely that, albeit in his own idiosyncratic way.
The inscription on Gilbert's memorial on the Thames Embankment in London reads: "His Foe was Folly, and his Weapon Wit." This is too coy. Gilbert saw the world as a chaotic place, in which our lives are brutal accidents of birth, fate and human blunder, a jungle of confusion and delusion, where we all aspire to be other than who we are, and where nobody is really who or what they seem to be.
Power. Status. Rank. Duty. Hypocrisy and affectation. Youth and old age. Gilbert's obsessions inform all these operas, his greatest being the arbitrary nature of society's absurd rules and regulations. He was a failed barrister in his youth and a lay magistrate in his old age. He loved the English legal world, not least for its theatricality, and he himself was compulsively litigious. But, for all his appearance as the very model of conservative respectability, his merciless lampooning of the heartless constraints of laws and etiquette reveal him, underneath it all, to have been a genuine free spirit and a true anarchist. Doubtless he would have denied these descriptions, but his subversive tendencies are beyond dispute, and he could hardly have been called a conformist.
The two principal elements of all the Savoy Operas are law and identity. Magic crops up in three of them, but material change caused by supernatural intervention is only a variation on the manipulation of laws and rules. There are love stories galore, but for the most part these do not drive the plot; taken out of context, they are sentimental and dull. As such, they are seldom distinguishable from the common fodder of ordinary light musical theatre, or indeed of Victorian melodrama.
If a key to understanding the operas is to see Gilbert as an anarchist, it may also be useful to approach them as the work of a proto-surrealist. With great fluidity and freedom, he continually challenges our natural expectations. First, within the framework of the story, he makes bizarre things happen, and turns the world on its head. Thus the Learned Judge marries the Plaintiff, the soldiers metamorphose into aesthetes, and so on, and nearly every opera is resolved by a deft moving of the goalposts.
But concurrently, Gilbert plays with different levels of reality, using para-theatrical conventions - that is, making characters refer implicitly to the fact that they are on a stage in a play, outside the framework of the audience's willing suspension of disbelief.
Am I alone,
And unobserved? I am!
Then let me own
I'm an aesthetic sham!
Read the whole of this confession by Bunthorne in Patience. Gilbert's joke, of course, is that Bunthorne is not unobserved - the audience is watching him. And he can only be talking to the audience. He is not discovering something new about himself before our very eyes, unlike the soliloquies of Hamlet or Macbeth, or even Malvolio, who must be played as real people talking to themselves, with total psychological truth. When Shakespeare needs to talk to the audience as such, he invents a non-character, such as the Chorus in Henry V.
The most extreme exercise in surrealism in this series is The Mikado, a puppet show cheerfully devoid of any sense of the real world as we know it. The far-fetched nature of its abstraction - it has nothing to do with Japan - and the craziness of its logic surely account for its being the most durable and popular of the operas. It abounds with para-theatrical devices:
Ko-Ko: Congratulate me, gentlemen, I've found a volunteer!
Chorus: The Japanese equivalent for Hear, Hear, Hear!
The Mikado himself observes: "It's an unjust world, and virtue is triumphant only in theatrical performances."
The Yeomen of the Guard is grounded in the real world and, as such, is the odd man out. It is probably a romantic rather than a comic opera, although it does contain some very funny moments, and the plot still involves disguise and mistaken identity.
"Comic operas" is what Gilbert and Sullivan called these shows. They are certainly not mere "light" operas, which are soft-centred romantic offerings; nor are they "operettas", which are frilly, frothy affairs, devoid of any shade of the dark side.
It is their dark side, their hard edge, that so distinguishes the Savoy Operas. Perhaps they may more usefully be described as grotesque operas. Or are they not absurd operas? Gilbert undoubtedly anticipated the Theatre of the Absurd, as did Alfred Jarry, whose Ubu Roi outraged Paris audiences in 1896. Ionesco's Rhinoceros is surely a magic-lozenge play, if a mournfully unfunny one, and it is no surprise that Samuel Beckett was a confirmed Gilbert & Sullivan aficionado.
The operas have often been misunderstood. They are referred to as satires, which they are not. There may be satirical elements in Iolanthe or Utopia Limited, but Gilbert's true intention is never to draw specific parallels. He merely holds up his mirror to the world, and reflects on its madness. Similarly misunderstood is his much-criticised attitude to elderly women. He is not attacking them; he is doing no more than lament the way life is. We all grow old, and the plain and the ugly have a harder time than the beautiful.
If these shows have fallen into disrepute over the years, it is because directors have failed to understand their raw edge. This results in boring, bland, sentimental, self-conscious, often gratuitously camp productions, which entirely miss the point.
What, then, is "Gilbertian"? The word has been in the English language for over a century, and to understand it we need to analyse the stylistic alchemy of Gilbert's art as a dramatist. His genius is to fuse opposites with an imperceptible sleight of hand, to blend the surreal with the real, and the caricature with the natural. In other words, to tell a perfectly outrageous story in a completely deadpan way. Indeed, to disguise a subversive anarchist bomb as bourgeois respectability.
But he was also a master of theatrical naturalism. Between 1863 and 1911, he wrote over 70 plays. Apart from his comic operas, translations of Donizetti and Offenbach, pantomimes and burlesques, a substantial proportion of his popular work consisted of naturalistic fourth-wall plays, all of them Gilbertian in their irony. These ranged from Engaged (1877), a farce about greed that still enjoys revivals today, to The Hooligan (1911), in which a condemned simpleton suffers a fatal heart attack in his prison cell.
Gilbert belonged to a small group of dramatists who reacted against the undisciplined melodramatic mess of the earlier Victorian theatre, so accurately evoked by Dickens in his portrayal of Mr Crummles and his company in Nicholas Nickleby. One of these was Tom Robertson, who was a major influence on Gilbert in two ways. As a dramatist, his so-called "cup-and-saucer" plays such as Ours (1866) and Caste (1867) depicted the world in a new, fresh, realistic way; and as a director (or stage-manager, as they called it in those days), he introduced Gilbert both to the revolutionary notion of disciplined rehearsals and to mise-en-scène or unity of style in the whole presentation - direction, design, music, acting.
Before this period, directing as we know it did not exist. On the continent, Wagner pioneered it at Bayreuth, as did the Duke of Saxe-Meiningen with his court theatre. In London, the actor-managers Macready, Kean and, later, Irving, organised their productions around their own egocentric performances. But it was Robertson and Gilbert who were the first real directors, and Robertson's early death in 1871 left Gilbert as the main pioneer of this new craft.
That Gilbert was a good director is not in doubt. He was able to extract from his actors natural, clear performances, which served the Gilbertian requirements of outrageousness delivered straight. He knew what he wanted - and he got it. And he was certainly a perfectionist. But whether his famously confrontational personality made him something of a dogmatic autocrat, as some of us modern practitioners in this field might question, is altogether another matter.
A further key to understanding Gilbert is to savour his talents as a visual artist. Had he never written a word, his work as an illustrator would have stood the test of time. All that survive in print are his many drawings for the Bab Ballads, the comic verses he wrote for Fun magazine in the 1860s. These have been published over the years in various collections, including, most recently, James Ellis's definitive Harvard University Press edition (1980). Gilbert added the Songs of a Savoyard - favourites from these operas - in editions from 1898, and in 2000 Jim Broadbent recorded a selection for Penguin Audiobooks, under my direction.
The Bab Ballads are masterpieces, which rank with Lewis Carroll and Edward Lear, and no appreciation of the Savoy Operas is complete without studying them. They contain the sources for many of the ideas, themes and plots of the operas. And within the Bab verses we find the two opposing strands of Gilbert's vision, the ridiculous and the real, the comic and the tragic. Gilbert was, above all, a great poet. It is easy to understand how he so inspired Arthur Sullivan.
Each of these men was an extremely skilled craftsman in his own right. But the key to their success is that they knew how to write for each other. For all their conflict, they had an extraordinary rapport. Despite their contrasting personalities - Sullivan ebullient and jolly, Gilbert sardonic and dour - they shared a sense of humour. Nothing else Sullivan wrote compares to the music provoked by Gilbert's inventive words. What he did to those words was to challenge and subvert them, and to enhance them by bringing out their flavour and meaning by interlacing and surrounding them with unpredictable succulent riches.
Much of Sullivan's serious and other lighter work is interesting and enjoyable. Without the Savoy Operas, he would certainly deserve a place in the pantheon of minor composers, not only for "The Lost Chord" and "Onward, Christian Soldiers", but for such pieces as his "Di Ballo" Overture, his Cello Concerto and his grand opera, Ivanhoe. The Bab Ballads alone would have earned Gilbert his place in posterity, although a dozen of his plays will stand up to revival when the world rediscovers them.
Nevertheless, it is obvious that, without each other, both Gilbert and Sullivan would probably have sunk without trace. Together, they created a unique body of work that places them alongside Molière, Congreve, Sheridan, Rossini, Donizetti, Offenbach and Feydeau - even the Mozart of Cosí Fan Tutte and The Marriage of Figaro. They paved the way for Wilde, Pinero, Shaw and Noël Coward; all the lyricists and composers of the great 20th-century American musicals cited Gilbert & Sullivan as their major influence and inspiration.
Sullivan's frustration was that he never had time to write proper music: he was convinced that he frittered away his life and his talents on the trivia of the Savoy Operas. How wrong he was.
These operas have been much discredited by stale productions. Some people simply loathe them on principle, courtesy of pompous elderly relatives and schoolteachers with awful singing voices. Others associate them with suffering the ignominy of playing members of the opposite sex in school productions of dubious motivation. But approached fresh, they are great shows - strong and youthful, and resonant with meaning. The Complete Gilbert and Sullivan, edited by Ed Glinert, is published on November 2 by Penguin at £30. To order a copy for £27 with free UK p&p, go to theguardian.com/bookshop or call 0870 836 0875. ENO's The Gondoliers opens at the London Coliseum, WC2 (0870 145 0200) on November 18.