Getting Started With Apple’s Vision Framework: A Developer’s Perspective | HackerNoon

The Vision framework was introduced by Apple in 2017 at WWDC as part of iOS 11. Its launch marked a turning point in the evolution of machine vision and image analysis, providing developers with native tools to analyze visual content and perform subsequent processing as needed.

In 2017, Vision introduced:

• Text recognition
• Face recognition
• Detection of rectangular shapes
• Barcode and QR code recognition

Since its debut, Apple has continuously enhanced the Vision framework, ensuring it evolves to meet modern requirements. By the end of 2024, with the release of iOS 18, Vision now offers:

• Improved text recognition accuracy with support for a large number of languages
• Detection of faces and their features
• The ability to analyze movements
• The ability to recognize poses, including the position of hands and key points of the human body
• Support for tracking objects in video
• Improved integration with CoreML for working with custom machine-learning models
• Deep integration with related frameworks, such as AVKit, ARKit

With the advent of the Vision framework, developers gained the ability to perform advanced image and video analysis tasks natively, without relying on third-party solutions. These capabilities include scanning documents, recognizing text, identifying faces and poses, detecting duplicate images, and automating various processes that streamline business operations.

In this article, we will look at the main scenarios of using Vision with code examples that will help you understand how to work with it, understand that it is not difficult, and start applying it in practice in your applications.

VNRequest

Vision has an abstract class VNRequest that defines data request structures in Vision, and descendant classes implement specific requests to perform specific tasks with an image.

All subclasses inherit the initializer from the VNRequest class.

public init(completionHandler: VNRequestCompletionHandler? = nil)

Which returns the result of processing the request. It is important to clarify that the result of the request will be returned in the same queue in which the request was sent.

Where VNRequestCompletionHandler is a typealias.

public typealias VNRequestCompletionHandler = (VNRequest, (any Error)?) -> Void

Which returns a VNRequest with the results of the request or an Error if the request was not executed due to some system error, incorrect image, etc.

The VNRecognizeTextRequest class from the abstract VNRequest class is designed to handle text recognition requests in images.

Example of implementing a request for text recognition:

import Vision
import UIKit

func recognizeText(from image: UIImage) {
    guard let cgImage = image.cgImage else { return }
    
    let request = VNRecognizeTextRequest { request, error in // 1
        guard let observations = request.results as? [VNRecognizedTextObservation] else { return } // 2
        
        for observation in observations {
            if let topCandidate = observation.topCandidates(1).first {
                print("Recognized text: (topCandidate.string)")
                print("Text boundingBox: (observation.boundingBox)")
                print("Accuracy: (topCandidate.confidence)")
            }
        }
    }
    
    request.recognitionLevel = .accurate
    request.usesLanguageCorrection = true
    
    let handler = VNImageRequestHandler(cgImage: cgImage, options: [:]) // 3
    try? handler.perform([request]) // 3
}

1. Create a VNRecognizeTextRequest for text recognition.

2. Receive the results of the text recognition request as arrays of VNRecognizedTextObservation objects.

The VNRecognizedTextObservation object contains:

• An array of recognized texts (VNRecognizedText().string)
• Recognition accuracy (VNRecognizedText().confidence)
• Coordinates of the recognized text on the image (VNRecognizedText().boundingBox)

3. Create a request for image processing, and send a request for text recognition.

4. Example: Recognition of tax identification number and passport number when developing your own SDK for document recognition

VNDetectFaceRectanglesRequest

This class finds faces in an image and returns their coordinates.

Example of implementing a face recognition request:

import Vision
import UIKit

func detectFaces(from image: UIImage) {
    guard let cgImage = image.cgImage else { return }
    
    let request = VNDetectFaceRectanglesRequest { request, error in // 1
        guard let results = request.results as? [VNFaceObservation] else { return } // 2
        
        for face in results {
            print("Face detected: (face.boundingBox)")
        }
    }
    
    let handler = VNImageRequestHandler(cgImage: cgImage, options: [:]) // 3
    try? handler.perform([request]) // 3
}

1. Create a VNDetectFaceRectanglesRequest for face recognition in an image.

2. Receive the results of the text recognition request as arrays of VNFaceObservation objects.

The VNFaceObservation object contains:

1. Coordinates of the recognized face VNFaceObservation().boundingBox.

2. Create a request for image processing and send a request for face recognition.

3. Example: In banks, there is a KYC onboarding where you have to take a photo with your passport; this way, you can confirm that this is the face of a real person.

VNDetectBarcodesRequest

This class recognizes and reads barcodes and QR codes from an image.

Example of implementing a request for recognizing and reading a barcode and QR code:

import Vision
import UIKit

func detectBarcodes(from image: UIImage) {
    guard let cgImage = image.cgImage else { return }
    
    let request = VNDetectBarcodesRequest { request, error in // 1
        guard let results = request.results as? [VNBarcodeObservation] else { return } // 2
        
        for qrcode in results {
            print("qr code was found: (qrcode.payloadStringValue ?? "not data")")
        }
    }
    
    let handler = VNImageRequestHandler(cgImage: cgImage, options: [:]) // 3
    try? handler.perform([request]) // 3
}

1. Create a VNDetectBarcodesRequest for recognition.

2. Get the results of the VNBarcodeObservation object array request.

The VNBarcodeObservation object contains many properties, including:

1. VNFaceObservation().payloadStringValue – the string value of the barcode or QR code.

2. Create a request for image processing, and send a request for face recognition.

3. Example: QR scanner for reading QR codes for payment.

We’ve covered the 3 main types of queries in Vision to help you get started with this powerful tool.