Harnessing CCTV image data for behavioural insight and modelling

Since the advent of digital cameras and image capture, a new world of behavioural and identity tracking has opened up, adding to the toolkit for data capture and analysis supporting the understanding and prediction of activity patterns

Cameras are these days watching everywhere: and what they see can increasingly become a major management tool. The subjects ‘under surveillance’ can be human – or mechanical. Early applications have involved vehicle and automatic number plate recognition (ANPR) allowing traffic patterns to be recorded and individual vehicle movements matched in origin and destination surveys and control regimes such as car parking, tolling and congestion charging.

Today, the use of human behaviour modelling and individual activity tracking through facial recognition is becoming a defined area of technical and professional development in its own right – with security, access and border control among the early adoption areas. Some elements are becoming defined in the specialist technology of biometrics – where very precise user-identification through iris recognition has been trialled for passport control, for example.

Where less individual-focused security or monitoring objectives apply, other applications have also been moving ahead. Queue management and crowd dynamics are two of the most advanced areas, particularly in stressed and capacity constrained environments like airports, terminals and stadia. One of the companies leading development in the this field, Amor Group, sees the prospect of 100 per cent people tracking in real time by 2020, using the fast-developing Google image network.

As well as video analytics from Blue Eye Video, the current Chroma ACDB system includes Bluetooth tracking from Blip Systems, thermal imaging people counters from Irisys, wireless tensabarrier counters from Qmetrix and ANPR vehicle tracking.

To date, Amor’s ChromaRMS systems include PAXPredict + Forecaster and PAXPlan + HMRS for staff management, which the company expects to integrate within its Chroma Airport Suite sometime during 2012.

It gives its ACDB ‘roadmap’ to full facial tracking as offering a less expensive way to capture passenger faces in real-time at the process start, and matching ‘in process’ faces at their end location. A scheduled release at the end of the year is planned.

Alongside this, Amor is incorporating additional tracking information from smartphone WiFi signals by passive tracking, with no requirement for an ‘app’. This is being released as an upgrade to BlipTrack, and integrates both WiFi and Bluetooth tracking data. Chroma’s ACDB system also involves thermal imaging path detection to cater for closely packed queues. Based on Irisys Thermal Imaging Technology, it is scheduled for release in the third quarter of 2012.

  Applied identity

Arguably the leading developer of face recognition systems is NEC. ‘It has been known for some time now that NEC has the most accurate facial recognition algorithms available world-wide,’ says Allevate, a UK-based company specialising in the field of “Applied Identity”. ‘This has been independently demonstrated by the US National Institute of Standards and Technology, and numerous other independent evaluations.’ NEC has

recently released a full suite of solutions incorporating this technology, with NEC Europe’s NeoFace Suite of face recognition solutions.

The solutions are designed to support and optimise surveillance, identification, security operations and people flows by using face recognition, focusing on law enforcement, government, border control and airport passenger environments.

But, say Allevate, there will undoubtedly be further application to numerous other environments as well, such as transport and retail.

NeoFace Match is specifically designed to allow authorities to match captured images against large databases of face photos. NeoFace Watch and NeoFace Find integrate with existing surveillance processes by extracting faces in real-time from CCTV and large volumes of video footage, and matching against a watchlist of individuals. NEC’s NeoFace Flow solution ‘timestamps’ individuals detected at camera locations to provide detailed information on people flows, such as average queue times. This provides insight on how passengers move through any environment. ‘No specific people identifying information need be recorded, and data can be purged at regular intervals,’ says the company

Video surveillance

Serious discussion of the technology and benefits of face recognition by cameras has been taking place at both meetings of the National Institute of Standards and Technology (NIST) and the IEEE Computational Intelligence Society in North America.

At this year’s NIST International Biometric Performance Conference in Gaithersburg, Maryland in March 2012, Dmitry Gorodnichy and Eric Granger of the Canada Border Services Agency explored the state of development of evaluation of real-time face recognition technologies for video surveillance applications.

Working in the Science and Engineering Directorate of the Agency’s Video Surveillance and Biometrics Section, they described the PROVIT project and how it evaluates state-of-the-art commercial technologies and academic systems for face recognition.

As well as public data sets for medium-to large-scale evaluation, there are evolving experimental protocols for different still-to-video and video-to-video surveillance applications, including screening of faces according to their resemblance to a wanted list, matching a face across several video feeds and fusion of face recognition from different cameras while tracking a person.

The Canadian project has looked at performance measures including transaction-based (P-R curve) and subject-based (biometric menagerie) analysis. This is an area where biometrics meets video surveillance, and evaluation of systems for Face Recognition in Video Surveillance (FRiVS) is looking at both publicly-available datasets and lab mock-ups to develop specialised performance metrics and protocols that integrate face recognition into operational CCTV environments.

Activity monitoring in the airport

The most active environment in which the relationship of CCTV and activity monitoring is being explored and applied is the airport.

French Specialist Blue Eye Video SA of Grenoble, which has been working in partnership with Amor, points out that increasing volumes of traffic and security constraints at airports have led to movement becoming increasingly inefficient, with passengers spending more time waiting to buy tickets, check in, at luggage inspection and at passport control. In this context, managing passenger wait time is crucial to improving the transport system’s performance, enabling visitors to spend more time in the retail areas and preventing costly delayed departures. Blue Eye believes its measurement and monitoring systems can help management anticipate traffic congestion problems and keep passenger traffic flowing as smoothly as possible, reducing wait times, optimising passenger flow and providing decision-making tools for better operator management.

The wider area of understanding passenger flow and behaviour now includes the use of data from counting sensors, people counting software, queue measurement, and wait time and database analysis tools. For image capture, an important step was using a video stream analysis combined with Sony Smart Camera architecture to determine how many people are waiting in queues and patterns of customer behaviour in a shopping, airport or stadium environment.

The provision of real-time data is a key element in ensuring that staff are deployed efficiently, Blue Eye says, enabling managers to make instant decisions and react to rapidly changing situations. The performance of different queues can also be compared and historical data used as an aid to resource planning by hour, day week or month.

Aéroports de Paris has been using Blue Eye Video queue monitoring for the past three years. The company believes that after six months its queue management solutions can dynamically optimise the resource allocation in relation to the number of passengers, enabling users to anticipate wait times beforehand.

Improving retail business revenues

Passenger experience is key to improving airport retail business revenues. By displaying the estimated time to reach each gate (taking into account queues at transfer and immigration), passengers can feel less stressed and more comfortable in taking up refreshment and retail offers without constantly watching the flight information display systems. Based on standard video cameras, the company believes four weeks’ observation is enough time to gain a real-time complete site overview.

When passenger queues encroach on passenger circulation areas, Blue Eye’s B-Queue system raises a real-time automatic alarm (SMS or email on a smartphone or tablet PC) with video proof, instantly alerting operation managers to a developing situation, with action plans implemented to optimize passenger flow and enhance security.

Blue Eye video was born out of INRIA, the French national research institution focusing on computer science, control theory and applied mathematics, and has developed new algorithms based on more than 15 years of research on image processing, and a focus on crowd behaviour, including the Muslim Pilgrimage in 2004 in Kingdom of Saudi Arabia for the Hajj.

The technology detects the density of people to stop more than 3 million people rushing during the event. In 2005, with Aéroports de Paris’ help by testing every three months, during three years Blue Eye Video created an innovative queue management solution. It now works at Charles de Gaulle airport, the sixth busiest airport in the world by passenger traffic, and also Dubai Airports with its partner Amor Group. It is also present in several countries with local partners, working with Delta Airlines (Atlanta), Washington Airports, Lisbon airports and also train stations in France (SNCF) and supermarkets in Europe (Carrefour group).

ChromaACDB is currently being applied as the world’s largest Service Delivery Measurement solution being delivered to Dubai Airport. At Oslo Airport, Chroma data aid analytics have been supporting passenger forecasting since 2009 and passenger tracking since 2010, with a terminal operations ‘dashboard’ currently being introduced.