Evaluating the information reliability of automated processes

Many auditors of financial statements use automated tools and techniques (ATT) in various aspects of their audits. ATT are IT-enabled processes that involve the automation of methods and procedures, such as the analysis of data using modelling or visualization. There could be significant implications for your audit if the information you use in your ATT is not sufficiently reliable.

This blog is for audit practitioners who use ATT. Continue reading to learn:

• the attributes that are relevant in considering or evaluating whether the information you are using in your ATT is reliable
• the factors that may assist you in assessing the importance of these attributes

With the evolution in technology, vast amounts of information and numerous types of information sources are available (e.g., social media, open data, blockchain, robotic process automation, Internet of Things, drones, etc.). The use of ATT allows you to incorporate information from various sources, both internal and external to the entity, and to use greater volumes in analyses. However, not all information that you use has the same degree of reliability.

When designing and performing audit procedures using ATT to obtain sufficient appropriate audit evidence, how do you consider reliability of information, including information produced by the entity?

Paragraph 7 of Canadian Auditing Standard (CAS) 500, Audit Evidence, requires you to consider the relevance and reliability of information to be used as audit evidence, including information from an external information source. When using information produced by the entity, paragraph 9 requires you to evaluate whether the information is sufficiently reliable for your purposes, including, as necessary in the circumstances:

• obtaining evidence about the accuracy and completeness of the information
• evaluating whether the information is sufficiently precise and detailed

Attributes affecting reliability of information

If the information you use in designing and applying your ATT is not reliable, that ATT will not provide a useful source of audit evidence. Let’s look at attributes of information reliability you could consider as you use an ATT. The attributes described below are:

• accuracy and completeness
• authenticity and credibility
• bias
• precision and detail

Accuracy and completeness

Accuracy deals with whether information is free from error in its reflection of the underlying conditions, events, circumstances, actions, or inactions, including the appropriate point in time. Completeness deals with whether information reflects all the relevant underlying conditions, events, circumstances, actions, or inactions.

Authenticity and credibility

Authenticity deals with whether the source actually generated or provided the information and was authorized to do so, and that the information has not been inappropriately altered. In considering whether the information is credible you may think about whether the source has the competence to generate the information and whether the source can be trusted.

Bias

Understanding bias in the information you are planning to use is an important input when considering potential for error or fraud as well as allowing you to exercise your professional skepticism. You may want to consider whether the information you are planning to use is free from bias, both intentional and unintentional, in how it reflects the underlying conditions, events, circumstances, actions, or inactions. While thinking about bias in the context of information you are planning to use as audit evidence, remember there is also potential for your own bias when using ATT, such as automation bias. Automation bias results from a tendency to favour output generated from automated systems, even when human reasoning or contradictory information raises questions as to whether such output is reliable or fit-for-purpose.

Precision and detail

Information needs to be sufficiently precise and detailed for the purpose of the audit procedure. In some cases, you may intend to use information produced by the entity for more than one audit purpose. Information may be sufficiently precise and detailed for one purpose but not be sufficiently precise and detailed for another purpose.

Factors affecting the relative importance of attributes

The above attributes and their relative importance, when considering the reliability of information to be used as audit evidence, may vary depending on a variety of factors. Further, not all attributes are necessarily relevant in all circumstances. Factors you may consider, along with related illustrative examples include:

The purpose of the audit procedure

Are you performing a risk assessment procedure or a further audit procedure (i.e., a test of controls or a substantive procedure)? You may use a two-way match to develop a report that compares an invoice unit cost with the related purchase order unit cost to identify differences, if any, over your predetermined threshold. If the output of the ATT is used as part of understanding the entity’s system of internal control in performing risk assessment procedures, you may determine that it is not necessary to evaluate the accuracy and completeness of such information, or it may not need to be as precise or detailed. However, if the output of the ATT will be used as part of a further audit procedure, you may determine that it is necessary to evaluate the accuracy and completeness of such information.

The source of the information

Is the information being accessed from an internal source or external source? You may use a model as part of your testing of an entity’s inventory that is subject to technological obsolescence. You might obtain information from an independent third-party, such as an industry association, regarding technological developments that may adversely affect the valuation of an entity’s inventory. Such information may be less prone to management bias, but its authenticity and credibility are likely important.

The nature of the information

Is the information quantitative or qualitative, and what is its level of detail? Information used in applying summary statistics to analyze average real estate rental prices for a country may not be sufficiently precise to use in performing further audit procedures to test revenue for a real estate entity. However, information used in the ATT to develop average real estate rental prices for a particular suburb may be sufficiently precise.

The controls over the preparation and maintenance of the information

What is your understanding of identified controls and your intended reliance on such controls? You may be using two and three-way matches combined with visualization to identify journal entries with unusual attributes related to nature, size, timing and authorization. If you determine that the controls over journal entries were designed appropriately and have been implemented, your work to consider the reliability of the journal entry information used in the ATT may be less extensive than when the entity lacks controls, or you have not tested the operating effectiveness of the controls.

How the information has been obtained and in what form

Did you obtain the information directly? Is the information from a customized report created by the entity? Did you have to perform manipulation of the data to get the information into a format that was usable for your audit? For your automated accounts receivable aging analysis, you may have extracted information directly from the subledger. In that case, your audit work regarding the reliability of this information may be less extensive than in circumstances when you have used balances extracted by management. As another example, you may have obtained information from an external source and merged it with internal information provided by the entity you are auditing. When merging or joining information, whether your process is properly planned to result in the precision and detail needed for your planned procedures may be an important consideration.

The relative importance of the procedure performed

Is the procedure being performed and the information used your primary source of audit evidence? You may have developed a model to provide your primary source of evidence regarding the entity’s provision for warranty liabilities. In establishing the reliability of the information underlying your model, including factors affecting likelihood of product returns and costs of meeting warranty claims, there may be a higher degree to which the attributes need to apply to that information as compared to a provision for warranty liabilities that has other sources of audit evidence that corroborate the conclusions and achieve the purpose of the audit procedure.

The assessed risks of material misstatement, and the reasons for the assessment

Is the risk of material misstatement to be addressed by your further audit procedure other than low? The entity’s revenue calculation may be subject to significant management judgment with many different contract provisions and possible different interpretations of when it is appropriate to recognize revenue. You have determined there is a presumed risk of fraud (and therefore a significant risk) of revenue overstatement due to possible management bias to overstate revenue. Therefore, because of the significant management judgment, the attribute of bias would apply to a higher degree compared to revenue that is recorded from deposits received.

In summary

In summary, professional judgment is needed in evaluating whether information to be used in an ATT is sufficiently reliable for your purposes. This blog provides possible attributes of reliable information and various factors that may assist you in complying with the requirements in CAS 500 to consider the reliability of information to be used as audit evidence. Depending on the degree to which the information intended to be used as audit evidence exhibits the applicable attributes, you may perform more or less work to conclude that the information being used as audit evidence is sufficiently reliable.

Keep the conversation going

Are you finding considerations of reliability of information challenging? What challenges are you most commonly running up against? With the exponential increase in emerging technologies and the expanding sources of information to be used as audit evidence, do you feel more is needed in CAS to enhance your ability to consider the reliability of that information? The International Auditing and Assurance Standards Board is currently revising ISA 500, Audit Evidence which CAS 500 is based on. If you have ideas about changes that may be needed in the standard, post a comment below or email me directly. I look forward to hearing your views.

Conversations about Audit Quality is designed to create an exchange of ideas on global audit quality developments and issues and their impact in Canada.

Disclaimer

The views and opinions expressed in this article are those of the author and do not necessarily reflect that of CPA Canada.