“Consider the final battle scene in the second of the three films, The Two Towers. In it, the heroic Rohirrim, a race of humans, make a desperate last stand against the hideous forces of Saruman, a dark wizard, inside a medieval-style fortress wedged into a narrow canyon.
In 2002, when The Two Towers was released, of course, audiences were accustomed to seeing vast crowds generated by special effects. But this was no ordinary movie trick, like those where groups of two hundred actors were digitally cut-and-pasted into a landscape over and over again. This was something new. In a way, Jackson had hired extras, only they were virtual ones rather than flesh and blood.
Each warrior in Saruman’s army–despite not being real–was moving on his own, guided by what he saw, heard, and felt during the battle. Each moved at his own pace, carried his lance in his own style, and was careful to keep from bumping into other warriors, even as they flowed around the giant rock. When the ladders went up, the warriors on the end of each one were virtual actors that Regelous calls “agents”. When others followed up the ladders, nobody, not even the movie’s director, was telling them what to do. They attacked knights in the castle in their own way.
‘When I say they have vision and hearing and sense of touch, I’m not talking metaphorically,’ Regelous explained. ‘They really do have vision, in that, on every frame of the movie, for every agent, an image was created by rendering the scene from their point of view. And those pixels actually got fed into the agent’s brain, where rules were applied to the pixels in that image.’
If an agent were programmed to fight, for example, the character might respond to an enemy nearby by turning toward him and swinging his sword. If he heard a loud nose, like an explosion, he’d probably look in that direction, like all the other agents around him. There was no telling precisely how he might behave, since his fuzzy-logic brain could respond in an almost infinite number of ways to the situation he was in, which made his actions seem only more lifelike.”
From The Smart Swarm, by Peter Miller