UI Component Detection for IBM AS400 System

🏢 The Problem#

IBM AS400 interfaces are decades-old, text-based screens with no modern API surface to hook into — there's no DOM to query, no accessibility tree, nothing a normal integration would use. The only way to know what's on screen is to look at it, which is exactly what teams had been doing manually: someone identifies which region of a screenshot is which UI component, by hand, every time a modernization project needed to map an AS400 screen to a modern equivalent.

🔬 The Decision#

This ruled out any code-level integration approach — it had to be computer vision. The less obvious call was not stopping at a CNN. A convolutional model alone is a reasonable baseline for object detection, but AS400 screens pack dense, often-overlapping UI regions into a small area, which is where CNN-based detectors tend to lose precision. I added Facebook's DETR (Detection Transformer) alongside the CNN pipeline specifically for that failure mode — transformer-based detection handles overlapping and densely packed regions more reliably than CNN-only approaches, at the cost of more training complexity than sticking with one architecture.

Noisy, inconsistent legacy screenshots were the other real problem: scan artifacts, inconsistent rendering, decades of accumulated UI drift. I addressed that with custom data augmentation tuned to legacy-interface variations rather than generic augmentation, annotating training data through CVAT.

🏗️ How It Was Built#

Two loops:

Train. AS400 screenshots get annotated in CVAT, run through custom augmentation for legacy-specific noise and layout variation, then used to train both the CNN and DETR models via transfer learning, with custom loss functions tuned for UI component classification rather than general object classes.

Detect. Live screenshots go through the same OpenCV preprocessing, then CNN + DETR inference maps detected components to their modern-UI equivalents in real time, feeding an integration API for downstream modernization tooling.

📈 Impact & Results#

• 85% detection accuracy on noisy, real-world legacy interfaces — the augmentation pipeline built specifically for AS400's visual quirks
• 60% less manual identification effort on modernization projects that previously required someone to map every screen by hand
• Two model architectures (CNN + DETR) working together to handle the dense, overlapping UI regions that a single-architecture approach struggled with