The Sensor Stack: How AVs Perceive the World
Before an autonomous vehicle can make a single decision, it needs to understand its environment — and that begins with sensors. Most AV systems layer three core technologies to achieve reliable perception.
Cameras provide rich visual data: color, contrast, text on signs, traffic light states, and lane markings. However, cameras struggle in low-light or glare conditions and cannot inherently measure distance.
Radar (Radio Detection and Ranging) compensates by measuring the speed and distance of surrounding objects using radio waves. It performs well in rain, fog, and darkness — conditions that degrade camera performance — making it a critical safety backup.
LiDAR (Light Detection and Ranging) fires rapid pulses of laser light to generate high-resolution, three-dimensional maps of the vehicle's surroundings. It gives the AV a precise spatial model of everything within its detection range. The technology has historically been expensive, though costs have fallen substantially as the industry has matured.
Most production AV systems rely on sensor fusion — combining and cross-checking data from all three sources simultaneously. This redundancy means a failure or degradation in one sensor type doesn't leave the system blind. Weather conditions expose the limits of this stack, particularly when sensors that typically compensate for each other are all degraded at once.
6
SAE automation levels defined
SAE International's J3016 standard, last revised in 2021, defines levels 0 through 5 to classify the degree of driving automation in a vehicle.
3
Core sensor types in most AV stacks
Cameras, radar, and LiDAR are the foundational perception technologies used in combination across most advanced autonomous vehicle programs.
Billions
Miles logged in AV simulation annually
Major AV developers use simulation environments to expose their systems to rare or dangerous scenarios that would be impractical or unsafe to replicate on public roads.
Perception, Planning, and Control: The Decision Loop
Raw sensor data is just numbers until software transforms it into meaning. Autonomous driving systems process information through a continuous three-stage loop running multiple times per second.
Perception is where sensor data becomes situational awareness. Machine learning models — typically deep neural networks — classify objects in the environment: pedestrians, cyclists, other vehicles, road boundaries, construction zones. The system also tracks how each detected object is moving and predicts where it will be moments from now.
Planning takes that situational picture and generates a path forward. This involves multiple layers: route planning (the broader journey), behavioral planning (should the car merge now, yield, or hold position?), and motion planning (the precise trajectory and speed to execute the chosen behavior safely).
Control translates the planned trajectory into mechanical commands — how much to steer, when to brake, how hard to accelerate. This stage must account for the physical dynamics of the specific vehicle.
The entire loop must handle edge cases — unusual scenarios that training data may not have fully anticipated — which is one reason developers log billions of miles of real-world and simulated driving to continuously improve model performance. Common misconceptions about self-driving often underestimate how difficult this loop is to make robust across the full range of real-world conditions.
Understanding What 'Supervised' Automation Means
If a vehicle's marketing describes features like lane centering or adaptive cruise control, those are driver-assistance tools — not autonomous driving. At SAE Level 2, the human driver remains legally and operationally responsible at all times. Treating these systems as autonomous can create dangerous inattention. Always read the owner's manual to understand exactly what a system can and cannot handle.
Levels of Automation and Where the Industry Stands
Not all self-driving technology is equal, and the industry uses a standardized framework to describe it. The SAE International J3016 standard defines six levels of vehicle automation, from Level 0 (no automation) to Level 5 (full autonomy in all conditions).
Most vehicles sold today with driver-assistance features sit at Level 2 — they can control steering and speed simultaneously, but the human driver must stay engaged and ready to intervene at any moment. A handful of systems operate at Level 3 in defined conditions, allowing the driver to disengage attention temporarily, though a handoff mechanism remains required.
Level 4 systems — capable of completing entire trips without driver input within a defined operational domain — exist in limited commercial robotaxi deployments in specific cities. Level 5, which would handle any road, weather, or geographic scenario with no human needed, remains an engineering goal rather than a current reality.
Regulatory approaches shape where and how each level can be deployed. Governments around the world are taking divergent approaches to authorizing AV operation, which directly affects where technology companies can test and launch services. Meanwhile, autonomous systems are finding early commercial footing in freight logistics — long-haul trucking is emerging as a primary proving ground before urban passenger applications scale widely.
Frequently Asked Questions
Most autonomous vehicles combine cameras, radar, and LiDAR (Light Detection and Ranging). Cameras capture visual detail like lane markings and traffic lights, radar detects objects and measures their speed in all weather conditions, and LiDAR generates precise 3D maps of the vehicle's surroundings by firing rapid pulses of laser light. Together, these sensors provide overlapping coverage that compensates for each technology's individual limitations.
Not in the broadest sense. Commercially available vehicles today range from SAE Level 2 (partial automation requiring constant driver supervision) to limited Level 4 deployments in defined geographic areas, such as robotaxi services in specific cities. True Level 5 autonomy — capable of handling any road, weather, or condition without a driver — does not yet exist in a production vehicle.
The vehicle's software continuously cycles through three stages: perception (interpreting sensor data to identify objects, road features, and hazards), planning (calculating the safest and most efficient path forward), and control (translating that plan into precise steering, braking, and acceleration commands). This loop runs many times per second and must handle unexpected scenarios in real time.
Adverse weather remains one of the hardest problems for AV systems. Rain and snow can obscure camera images and scatter LiDAR pulses, reducing detection range and accuracy. Engineers are developing sensor fusion techniques, higher-resolution LiDAR, and machine learning models trained on weather-specific data to improve performance, but challenging conditions still require extra caution or operational restrictions.
Autonomous vehicle testing and limited commercial deployment are happening in several U.S. cities, parts of Europe, China, Singapore, and elsewhere. Freight trucking on defined highway corridors is also an active proving ground. Regulations, geography, and infrastructure quality vary widely, meaning AV deployment is uneven across regions.
The content on this site is for informational purposes only and is not a substitute for professional advice. Always consult a qualified professional for guidance specific to your situation.