Everyone knows what generative AI is for, don’t they?
When generative AI is performed in a data center, with access to practically unlimited compute and power resources, it is the modern miracle which provides A+ answers to examination questions, generates ‘photographs’ which look like images taken by a real camera, and predicts molecular behavior that previously would have needed to be observed in a laboratory experiment.
What about generative AI at the edge and endpoint though? The technology is incredibly exciting.
But what exactly is it for?
This question was once of only hypothetical interest, but now, it’s affecting the decisions of embedded device developers for current design projects. That’s because the launch of the new generation of Ensemble microcontrollers and fusion processors has given developers for the first time a hardware platform which supports transformer operations, the key software function which underpins generative AI models.
The low power consumption and fast inference performance of the new Ensemble E4, E6 and E8 products make the implementation of generative AI a real possibility in endpoint devices, even those powered by very small batteries. It’s a big move for the AI market: the large language models which enable generative AI services such as Gemini and Perplexity run on huge arrays of large, expensive, power-hungry GPUs in data centers.
Bringing generative AI within the scope of battery-powered systems
Now, generative AI can run locally in small, battery-powered embedded devices for the home, office, factory or smart city – and it does not need an array of GPUs. Instead, Alif has assembled in the Ensemble devices the features which enable generative AI at the edge:
- An NPU, the Arm® Ethos™-U85, which can perform transformer operations and other neural networking operations at high speed and low power
- Tight integration of the NPU with the CPUs – Arm Cortex®-M55 in the dual-core Ensemble E4 and Cortex-M55 plus Cortex-A32 in the E6 and E8 fusion processors
- A wide memory subsystem for extremely fast on- and off-chip transactions
- Support for up to two MIPI-CSI image sensors, alongside a fully hardware-accelerated image signal processor pipeline which operates at up to 60fps at 2MP resolution, enabling high-speed vision AI processing
This hardware platform makes generative AI possible at amazingly low power: just 36mW when executing a small language model (SLM) to generate text on an Ensemble E4, for instance.
Where natural language capability adds value
So with generative AI now feasible at the edge and endpoint, developers are starting to figure out what to do with it. Some types of devices obviously lend themselves to the use of generative AI: in smart glasses, for instance, real-time translation of a foreign language is a valuable use case, as well as understanding natural language instructions or queries from the user. And in a smart security camera, generative AI could be used to produce context-aware spoken warnings to a potential intruder. (‘If you’re the adult male in the black hoodie who has been hanging around for the past five minutes, you should know that I have taken video footage of you and uploaded it to the local police department.’)
Discovering what the developer’s imagination can invent
So there are some natural-language functions which can obviously supplement the operation of existing types of products.
But the capabilities of generative AI extend far beyond language. What might the possibilities be in embedded devices?
All that we know at this stage is that many potential uses of generative AI at the edge are yet to be discovered or imagined. This is completely natural, given that the potential to deploy gen AI at the edge and endpointhas only been apparent to developers for a matter of months. But when new generative AI use cases at the edge do emerge, one thing is almost certain: they will be running on an Ensemble device.