Natural language processing trials

In February 2018, ASIC released a set of problem statements with use cases to understand and encourage the application of natural language processing (NLP) in resolving regulatory problems. The trials were to explore potential efficiencies in supervision, including through automation and prediction, and provided a genuine learning opportunity for ASIC. ASIC’s tender was issued for the provision of pilots in:

• Identifying promotions of concern for financial and credit services
• Phone sales practices of insurance
• Managed-fund Product Disclosure Statements (PDS) review
• Financial advice file review
• Financial reporting review of company announcements
• Prospectus review.

ASIC’s 2018 NLP trial results

Data availability

NLP does not transform poor-quality data into high-quality data, and it cannot find data where none exists. Therefore, a challenge in ASIC’s NLP trials has been sourcing the right kind of data – and in sufficient quantity. Poor quality or inadequate data produces models that do not produce useful outputs or that are very limited in scope.

Data annotation

For machine-learning processes ASIC needed to ‘teach’ the machine what it needed to find. This included annotating PDS’s to highlight relevant features, which was time consuming, as was the time required to identify problematic examples.

This led to a second issue – a lack of sufficient quantity. ASIC needed a large volume of misleading or deceptive examples to teach the machine what to look for. These example are rare.

Next steps: enhancing our knowledge base, personnel and capacity

Like many organisations, ASIC is considering the skills and resources it needs to integrate regtech into its everyday operations. The NLP trials provided practical insight on that front.

ASIC is conducting further NLP-related trials in 2019. See the events section for more information.