Can’t wait for the cloud? In the fields of manufacturing, medicine, and robots even tiny delays can change the next move, so where you crunch is just as critical as how smart the model.
Edge AI is the on-device or near-device deployment of artificial intelligence so that data can be analyzed and processed closer to where it was created in order to reduce network latency, minimize the transmission of extraneous data and respond to situations when a network connection is not available.
But operating AI at the edge is about more than having a smaller model. In this piece, see the edge AI hardware, software, and data-processing tools, how edge fits in the cloud, and real-world examples of where you can apply it.
Why Now? The Drivers Behind Edge AI Adoption
More and more data are being created by connected devices and sensors, which is stressing cloud AI. Factories, hospitals, retail stores and logistics centres all have constant streams of images, videos and measurements from sensors. For data-driven companies dealing with video datasets, all frames should be transferred into the cloud to be processed, which could greatly affect bandwidth pricing and make data transfer slow.
On the other hand, advances in 5G connectivity, AI accelerators that became smaller and efficient neural networks allow local processing to become more possible. 5G can provide communication between systems that are connected, while edge AI does not require any connection at all. And quantization enables models to fit within the memory and power budgets of smaller devices.
These advances have made edge AI a viable solution for enterprises, that requires quicker decisions and has lower data transfer requirements along with increased resilience in the operations.
Inside the Edge AI Stack
How does Edge AI function? This technology is based on the use of small-sized AI models, as well as hardware and software that work on the analysis of information that is gathered locally. There are different responsibilities of each component. Devices gather the data, while AI inference is carried out on local processors.
On-Device Models
Local running of AI involves using smaller models that have to fit in the available memory space, computational abilities, and energy. SLMs allow selected language tasks to be performed with less demand for resources compared to the larger ones. Quantization is an approach which minimizes the precision of the numerical representation of model parameters and may contribute to memory optimization and efficient inference.
The development of compact AI models is evident from Google’s EmbeddingGemma 2. The 740-million-parameter multimodal embedding model is meant for on-device inference. On the Pixel 11 Pro, the quantized text-only weights occupy about 191 MB of active RAM. As this model belongs to the category of embedding models, it means that it does not have the function of a chatbot but just helps perform specific tasks like semantic search and retrieval.
Specialized Hardware
Software alone cannot deliver efficient edge inference. They are responsible for performing the calculations that make up an AI model, with a consideration for speed, power, and heat, all of which must be kept within physical and/or power constraints. These are neural processing units (NPUs), mobile GPU cores, and dedicated AI accelerators. Data-center processors are designed to handle those workloads at enormous scale.
Nvidia says its Jetson Orin Nano 2 offers twice the inference performance of its Jetson Orin Nano in the same form factor. The 15-watt version consumes 40% less power to achieve the same performance. All that can make a difference for robots, smart cameras and industrial systems where local AI processing is needed but on a smaller footprint and at a lower power level.
Real-Time Data Processing
Edge gateways link the sensors and devices with local AI processing and the cloud services infrastructure. The edge gateways are capable of aggregating sensor data from different sources, performing inference, filtering trivial events, and passing along the valuable information for analysis. A device monitoring system can process the vibration sensor data at the edge and raise an alarm only when the model recognizes an anomaly.
The architecture below shows how these layers work together.
Also Read: Cloud AI vs. Edge AI: How Should Enterprises Split the Workload?
The Hybrid Intelligence Model
Both cloud AI and edge AI deliver value to an enterprise, and the two work best when used together. The cloud is designed to perform heavy computations, including training large foundation models, aggregating data from the remote locations for analysis, and storing historical data for analysis. Edge AI performs inference closer to where the data is generated.
For instance, factory equipment can identify faults on the fly with edge AI and transmit the crucial data it generates to the cloud for analysis and model retraining. This provides local agility along with the strength of cloud computing.
Microsoft’s Azure Local supports AI inference on local infrastructure using CPUs or GPUs, including in disconnected environments. It combines local processing with cloud-consistent management, although some documented AI workloads remain in preview.
| Factor | Cloud AI | Edge AI |
| Latency | Depends on network conditions and remote processing | Typically lower for local inference |
| Bandwidth | May require substantial data transfers | Reduces transfers through local processing |
| Privacy | Data may travel to cloud infrastructure | Data can remain on local systems |
| Cost | Includes compute, storage, and transfer costs | Includes hardware, deployment, and maintenance costs |
How much to put in the cloud versus edge will vary by workload, connectivity, and privacy needs, as well as cost of infrastructure. But in most enterprise use cases, the optimal approach is to treat cloud and edge as brothers, not rivals.
Strategic Enterprise Benefits
Businesses use Edge AI to operate more quickly, efficiently process data and maintain important functions when their internet connection is unstable. In locations where delays, excess data or system outages impact safety, quality of service and effectiveness, Edge AI is especially critical.
Faster Decisions at the Point of Action
Some autonomy-enabled equipment such as autonomous vehicles, industrial robots and automated lines have to operate on tight timescales. Taking all inputs from each device back to a cloud-based server leaves delays in the network that can affect critical decisions. Edge AI powers the capability to analyze outputs at the equipment site itself, allowing a machine to sense something in its way, diagnose an anomaly or adapt its operations without having to go to a distant data center. An additional 77% of respondents to an Intel-commissioned survey believe that real-time decisioning is crucial to many industrial AI workloads.
Greater Control Over Sensitive Data
Local processing may help minimize the amount of PII that gets shared with external systems. Healthcare providers, banks, and other businesses covered by regulations will be able to perform analysis of sensitive information in closer proximity to the source, thus limiting its exposure during transit. Processing data locally will not guarantee compliance with regulations like HIPAA or GDPR, however.
Resilience When Connectivity Fails
Network interruptions will cause loss of access to necessary services on cloud-based systems. With edge AI, one can be assured that specific processes will continue in an uninterrupted manner within the device even without internet connection since devices can detect anomalies and issue alarms based on prearranged response patterns.
These two advantages make edge AI a necessity in any setting where immediate decisions, proper data management, and operation have an impact on organizational performance.
Real-World Enterprise Use Cases
Edge AI can be valuable for any business that has to analyze data quickly and respond to it at or near the source. Here’s how local inference can meet various operational needs in manufacturing, healthcare and retail.
Manufacturing
The factory can employ acoustic sensors to collect audio and vibration signals generated from the industrial machinery. Using an Edge AI algorithm, these signals are analyzed for detecting any anomaly that could suggest any wear or fault. This assists the maintenance team to investigate these issues even before they result in costly machine downtime. As noted in the AWS example of pump monitoring, a quantized ONNX model is used to detect the anomalies in the edge gateway, whereas difficult cases can be forwarded to the cloud agent.
Healthcare
A wearable device could analyze its own electrocardiogram (ECG) signal to identify particular patterns, which could be signs of potential heart rhythm disturbances. Rather than transmitting the data for every reading it takes, the device could identify and flag a dangerous pattern and send a warning to the wearer. These systems would still need valid clinical validation, though, and they shouldn’t automatically be considered a diagnosis.
Retail
It is possible for retailers to make use of edge AI and computer vision for the identification of products in smart check-out stations, checking shelf availability, and keeping track of inventory movement. By processing data locally and not having to stream video continuously to the cloud, it is possible for the central system to aggregate insights from select stores.
The approach remains the same throughout these industries: Process time sensitive data locally while using the cloud where there is added benefit from doing so.
The Road Ahead
AI’s future will be driven by powerful cloud infrastructure coupled with intelligence at the edge. As the edge hardware continues to become more sophisticated and optimized, business will be able to act faster, eliminate duplication of data and continue to operate in the event of a network outage. The trick will be deciding which workloads to run locally and not moving everything to the edge no matter what.
IT executives must begin by examining the current data pipelines and identifying the workloads that require immediate processing and the possibilities for local inference to mitigate cloud computing and data transportation expenses. All this to create a smarter, more streamlined AI architecture that is operationally focused.
