The system is basically a smart camera that watches a crosswalk and logs every time a pedestrian crosses or a vehicle passes through. It uses two AI models working together: the IMX500's built-in nanodet_plus model for general detection (especially good at vehicles), and YOLOv8n for enhanced pedestrian detection.

The key innovation is the dual-model pipeline. The IMX500 alone would miss pedestrians who were more than 15 feet away because they'd be too small in the frame. So I extract just the crosswalk area, upscale it 2.5x, and run it through YOLO. Then I merge the detections using IoU (Intersection over Union) to avoid counting the same person twice.

Everything runs in real-time on the Raspberry Pi, with a web interface at localhost:8080 showing live detections. The system logs events to separate files for pedestrians, vehicles, and combined events, all in JSONL format with timestamps and confidence scores.

The IMX500 is pretty special. It's a camera module with an integrated neural processing unit, meaning it can run AI models directly on the sensor itself. This is way more efficient than sending every frame to the Raspberry Pi's CPU for processing. The IMX500 handles the heavy lifting of running nanodet_plus on every frame.

For field testing, I'd set everything up on the sidewalk with a portable power bank. The trickiest part was getting the camera angle right. You need to see the entire crosswalk clearly, which sometimes meant I'd bring a small ladder to get elevation. Nothing says "I'm definitely not doing anything suspicious" like a teenager with a ladder and computer equipment on the side of a highway... I was asked a couple times by passersby's what I was doing, and I was so afraid that they'd think I'm planting a bomb on campus and call the cops on me.

Here's the problem I discovered during testing: when a pedestrian is 15+ feet away from the camera, they're only a few dozen pixels tall in the frame. The IMX500 running nanodet_plus would just... miss them. It would pick them up once they got closer, but by then you've lost valuable data about when they entered the crosswalk.

My solution was ROI (Region of Interest) upscaling. I defined the crosswalk area as a polygon in a JSON config file. For each frame, I extract just that area, upscale it 2.5x using nearest-neighbor interpolation (fast and good enough for this use case), and run YOLOv8n on the upscaled region. Now those distant pedestrians are big enough to detect reliably.

But this creates a new problem: now both models might detect the same person. That's where IoU-based merging comes in. For each pair of detections, I calculate their Intersection over Union. If IoU > 0.5 and they're the same object type (both "person" or both "car"), they're probably the same object, so I keep only the detection with higher confidence.

The result: reliable vehicle detection from the fast IMX500 model, enhanced pedestrian detection from the YOLO upscaling pipeline, and no duplicate counting.

This was genuinely one of the most annoying bugs I encountered. Picture this: there's a car parked 20 feet from the crosswalk. The detector sees it in frame 1, doesn't see it in frame 2 (maybe lighting changed, maybe the confidence dipped below threshold), sees it again in frame 3, doesn't see it in frame 4... you get the idea.

My naive tracking system would log a "vehicle entered zone" event every single time the detection flickered back on. One parked car over 30 seconds could generate hundreds of bogus log entries. I called them "ghost cars."

To fix this, I used distance-based cooldown tracking. The system now maintains a history of the last 10 positions for each tracked object. When a new detection comes in, I check: is this detection more than 100 pixels away from the last logged position for this object? If yes, it's genuinely a new object (or the same object has moved significantly). If no, it's probably just the detector flickering, so I ignore it until the cooldown expires.

This single change dropped false positive logging by a lot. The key insight was that real crosswalk events involve motion. A car entering the crosswalk moves at least several feet between frames. Detector flicker happens in place.

Bounding boxes are rectangles drawn around detected objects. The obvious way to check if an object is in a zone is to check if the center point of its bounding box is inside the zone polygon. This works great for pedestrians.

But for vehicles? Not so much. A car's bounding box includes the entire vehicle from the ground to the top of the roof. The mathematical center of that box is somewhere around the roof line. But the part of the car that actually matters for crosswalk safety is the front bumper, which is at the bottom of the bounding box.

So I'd get situations where a car's front bumper was clearly inside the crosswalk zone, but its "center point" (up at the roof) was still outside the zone boundary. The system wouldn't log it as being in the crosswalk. Terrible.

The fix: instead of using the center point (50% down the bounding box), I use a weighted point at 75% down. For vehicles, this puts the detection point much closer to where the bumper is, which is the part we actually care about. For pedestrians, it doesn't matter much since they're roughly vertical.

Defining zone boundaries by manually typing in coordinates would be absolutely miserable. You'd have to guess pixel coordinates, run the system, see if the zone looks right, adjust coordinates, repeat. No thanks.

Instead, find_borders.py launches a web interface showing the live camera feed. You click on the corners of your zone (crosswalk, sidewalk, vehicle lanes, etc.) in order, and it displays the polygon in real-time as you build it. Once you're happy with it, you save it to a JSON file with a descriptive name.

The JSON files live in the zones/ directory and look like:

The main detection script loads these zone definitions and checks which zones each object is in every frame. You can easily switch between different zone configurations by loading different JSON files, which is great for testing different camera angles or monitoring different crosswalks.

I used Claude Code (Anthropic's AI coding assistant) extensively for this project. The workflow went something like:

1. Field test - Set up the system at the actual crosswalk and watch what happens
2. Identify problems - "Why is it logging the same parked car 50 times?" or "Why is it missing pedestrians?"
3. Design solution - Figure out the algorithmic approach (distance-based cooldown, ROI upscaling, etc.)
4. Implement via Claude - Have Claude write the Python code to implement the solution
5. Debug and refine - Test the changes, identify edge cases, iterate

Claude wrote almost all of the code of the code, but all the problem-solving and creative solutions came from my side. The AI is excellent at implementing well-specified algorithms and handling the tedious parts of Python syntax, OpenCV calls, and proper error handling. But it can't tell you that you need distance-based cooldown or weighted detection points. That requires actually understanding the domain problem.

analyze_crossings.py
This script reads the JSONL log files and generates statistical reports: How many pedestrians crossed per hour? What's the peak traffic time? What's the ratio of vehicles to pedestrians? How often do vehicles enter the crosswalk while pedestrians are present? This is the tool that actually turns raw detection logs into actionable data for advocacy.

record_video.py
Sometimes you need to see what the system saw. This records video with the zone boundaries and detections drawn on each frame, which is incredibly useful for debugging. You can watch the recording and see exactly when and why the system made certain decisions.

photo_grid.py
Takes a snapshot from the camera with a coordinate grid overlay. Super useful for planning zone layouts before you go into the field. You can look at a gridded photo and sketch out roughly where your zone boundaries should be, then use find_borders.py to click the exact points.

Current Status
The system works...? I've deployed it at the crosswalk for a bit, collected data, and confirmed that the detection rates are solid. The dual-model pipeline catches both distant pedestrians and vehicles reliably. The ghost car problem is solved. The weighted detection points correctly identify when vehicles enter the crosswalk zone.

Future Improvements (from my todo.txt)

• Directional detection - Add sidewalk zones on all four sides so I can tell which direction pedestrians are traveling
• Complete crossing sequences - Track the full path: sidewalk east → crosswalk → sidewalk west, with timestamps for entry and exit
• Crossing duration - Calculate how long each crossing takes from entry to exit
• Better tracking consistency - Maintain tracking IDs more reliably across zone transitions
• Analytics dashboard - Build a web dashboard for visualizing patterns, peak hours, and safety metrics

The Big Goal
Eventually, I want to take all this data back to the town and state government with actual evidence: "Here's proof that X pedestrians cross during this time window, Y vehicles enter the crosswalk while pedestrians are present, and the current signal timing causes Z conflicts per hour." Real data is harder to ignore than complaints.