Alberta Reliability Standards Compliance Working Group Meeting June 19, 2012 Minutes are noted on the summary slide for each section Agenda Topics 1. Introduction 2. Presentation of discussion plan, method and scope 3. Presentation on compliance monitoring audit process issues 4. Initial discussion on solutions 5. Meeting debrief Plan вЂў Review feedback and observations obtained from previous yearвЂ™s audit experience вЂў Provide input on possible solutions to identified process issues вЂў Timelines вЂў 1st meeting (today) вЂў CWG review issues вЂў Gather input from CWG вЂў 2nd Meeting вЂ“ DRAFT (end July or early August) вЂў AESO presents solutions based on first meeting input вЂў CWG refines solutions вЂў 3rd Meeting вЂ“ FINAL (End August or early September) вЂў AESO presents updated solutions as per 2rd meeting refinements вЂў 4th meeting if required (End September or early October) вЂў Publish guidelines вЂ“ one month following final meeting Method, Scope Method вЂў Go over the issues in order of the compliance monitoring audit process sections (guideline process map) вЂў As a group: discuss each issue focusing on: impact, broadness and provide input on possible solutions вЂў Feedback will be used by the AESO to update the audit guideline Scope вЂў Scope is limited to: вЂў The compliance monitoring audit process вЂў Existing, clearly defined issues. вЂў Existing constraints (legislation and mandate) вЂў Issues outside of the scope will be tabled and brought forward at a later date or other forum. Process sections Identified Compliance monitoring audit issues (13) (numbers 7.x refer to the audit guideline sections) Process Section Process Section Issues Issues 3 7.7 Review for completeness 3 Publish audit schedule 0 7.8 Assess audit evidence 0 7.2 Changes to audit schedule 1 7.9 Issue draft report 0 7.3 Workshop /Notification 3 7.10 Post-audit conference 0 7.4 Pre-audit conference 0 7.11 Review draft audit report 1 7.5 Gather audit evidence 0 7.12 Issue final audit report 0 7.6 Submit audit evidence 2 7.13-14 Referral to MSA and retain evidence 0 GEN Information / Guidelines 7.1 Defined Issues General: Information/Guidelines 1. Applicability Assessments Participants/CMP - Process change to reflect updated applicability sections - Complete 2. Acknowledgement and Consent Participants/CMP - Simplification to meet business need - End August 3. Guideline overly complex - Need to define specific issues - CWG Input Participants Defined Issues Audit Guideline - 7.2: Change to audit schedule 4. Request for deadline extensions - Requests for deadline extensions were unreasonable - CWG Review (from December 2011 CWG) CMP Defined Issues Audit Guideline - 7.3: Workshop / Notification 5. Lessons learned and timing Participants - When can lessons learned be shared? - CWG Input 6. Publish generic index template Participants - Can a generic index be published in advance? - Related to #9 7. Asset identification - Need to identify an asset list prior to audit - CWG Input Participants/CMP Defined Issues Audit Guideline - 7.6: Submit audit evidence 8. Bad index of evidence files CMP - Difficulties in finding evidence - Created standard index template (Attachment 4) - Related to #9 (from December 2011 CWG) 9. Audit index VS RSAWs Participants/CMP - What method can be used for indexing evidence? - CWG Input Defined Issues Audit Guideline - 7.7: Review for completeness 10. Initial review too detailed CMP - Some assessment was done during completeness review - CWG Review (From December 2011 CWG) 11. How can we help entities better manage information requests? Participants/CMP - Should we proposed tentative timelines at this stage? - CWG Input 12. Additional request content is sometimes unclear Participants - Requests are not always clear resulting in multiple follow-up - CWG Input Defined Issues Audit Guideline - 7.11: Review draft audit report 13. Re-issue of draft report - Extent of commenting on re-issued draft report is unclear - CWG Input CMP Additional issue brought forward by a participant NEW: Difficulties with overlap of self-certification and audit - What are the consequences of self-certifying compliance if there is a subsequent draft audit report showing a suspected contravention? - Discussion was held at the meeting with follow-up at the next. Discussion on issues WORKSHEETS Each worksheet will show: 1. Title 2. Background 3. Current status 4. Possible constraints After a quick review of the issue we will begin discussions: 1. Gather thoughts on impact and broadness 2. Discuss possible solutions Discussion on issues SUMMARY At the end of the discussion, we will: 1. Write a summary of the main points for all to see 2. Validate that the summary contains all pertinent discussions NOTE: It is important that we validate the summary as these points will be used to create the solutions that will be presented at the 2nd meeting. General 3 Guideline overly complex Background During an end of compliance monitoring audit survey a market participant stated that the guideline was not used because it is overly complex. Current status вЂњOverly complexвЂќ is unclear and we need input Possible Constraints The purpose of the guideline is to define the procedures for planning, executing, and reporting with respect to each compliance monitoring audit. It has the dual purpose of providing policy/rules and procedures (вЂњhow toвЂќ). During a compliance monitoring audit one is more likely to simply look for the procedure which is currently mixed with the policy/rules. It is possible to change the guideline however, both elements must remain. 3 Audit Guideline overly complex Our thoughts The guideline(s) could be changed as follows: 1. Separate policy/rules from the вЂњHow toвЂќ 2. Create a simplified вЂњhow toвЂќ section 3. Use FAQs for missed items - Update annually with FAQ transfer to guideline 4. Roll-out update to all participants (newsletter) 5. Send to new participants Group discussionвЂ¦ What are the issues, impact and broadness? 3 Audit guideline overly complex SUMMARY The CWG: вЂў Agrees with the approach to split the вЂњhow toвЂќ and policy вЂў There is always room for improvement in the audit guidelines Audit guideline - 7.2: Change to audit schedule 4 (Review) Requests for deadline extensions were unreasonable Background Some MPs requested extensions of all or most deadlines and did not provide sufficient reasons. Guideline states вЂњreasons beyond the market participantвЂ™s controlвЂќ which is not clear. Most reasons were related to internal scheduling or change of staff. Current status As discussed at the December 2011 CWG: вЂў The guideline is looking for вЂњexceptionalвЂќ reason. вЂў Entity needs to build a case вЂў We will communicate intent through compliance audit training sessions вЂў We will change guideline to provide better guidance Audit guideline - 7.3: Workshops / Notification 5 Lessons learned and timing Background We have been asked by entities to share lessons learned. вЂњLessons learnedвЂќ are shared during our audit training sessions - however, this is only useful to scheduled participants which may not be timely for all. Current status вЂњLessons learnedвЂќ are provided during the audit training of a scheduled entity. Possible Constraints вЂњLessons learnedвЂќ cannot be identified in the audit report - under rule 12, only suspected contraventions are identified. 5 Lessons learned and timing Our thoughts As discussed at the December 2011 CWG meeting: вЂў We have modified training to focus on market participant needs вЂў Emphasis is placed on key areas (index, pointing to evidence, use of attestations and relating evidence to requirements/measures (location within a file) Further to this, we will: вЂў Include a вЂњlessons learnedвЂќ module at the Stakeholder Information Sessions вЂў Provide a review of expectations at RSAW training вЂў New: Ad hoc вЂў Old: Request stakeholder input on interest and timing Group discussionвЂ¦ Does this work? 5 Lessons learned and timing SUMMARY The CWG: вЂў Agrees that a module at the stakeholder information sessions for lessons learned on the general audit process would be valuable вЂў There is a need for RSAW sessions to go over auditor expectations: вЂў Start with statistically problematic вЂў When possible group by standards areas вЂў Possibly include AESO technical SME at the session вЂў Suggest once a year for problem standards Audit guideline - 7.3: Workshops / Notification 6 Publish generic index template Background Entities are requested to fill out the index template when submitting audit evidence. The index is sent out as an attachment to the notification letters. Some entities requested that a copy of the index template be published well in advance of the audit. Current status We provide a copy filtered by entity type as an attachment to the notification letter. A generic (non-filtered) copy is provided upon request. Possible Constraints This issue is related to #8 (difficulties for auditors in finding evidence) and with #9 (use of RSAWs or Index) 6 Publish generic index template Our thoughts We have no issues with publishing a generic index вЂ“ however, since the need for this depends on our decision to use RSAWs or not, we propose that 6, 8 and 9 be discussed as one issue. Group discussionвЂ¦ Does this work? 6 Publish generic index template SUMMARY The CWG: вЂў Agreed to discuss this issue with issue #9 Audit guideline - 7.3: Workshops / Notification 7 Need to identify an asset list prior to audit (Can extend to relationships and other information) Background The need for information depends on individual standard requirements and on entity structure. As an example, a participant requested that we provide a list of assets that will be included in the audit. Current status Information is requested and obtained during the audit. Possible Constraints Since the need for information varies by standard and by entity structure, requests for lists cannot be generic. As an example, an entity may have different CIP procedures for different plants within their organisation вЂ“ in such a case, a list of operating personnel is required for each plant (the asset) so that the yearly provision of the procedure can be verified. 7 Need to identify an asset list prior to audit? Our thoughts Lists are not limited strictly to assets. The lists are required to perform certain assessments. Requests depend on individual standard requirements and entity structure which may include but is not limited to: вЂў Functional entity type (who owns, who operates and who performs PP function) вЂў Asset list to verify inclusion or exclusion Possible solutions: вЂў Introduce pre-audit questionnaire вЂ“ similar concept to WECC. вЂў More information could be requested in the RSAW. Group discussionвЂ¦ Impact (timing), broadness, alternatives? 7 Need to identify an asset list prior to audit? SUMMARY The CWG: вЂў Is in favour of using a pre-audit questionnaire вЂў There will be a possible need for training Audit guideline - 7.6: Submit audit evidence 8 Bad index of evidence files Background Based on the 2011 Q2 and Q3 audit experience, it was found that the suggested index format as described in appendix B of the audit guideline, did not provide clear guidelines on what was required. Further to this, entities were allowed to submit a variety of index styles. Having a variety of index structures led to difficulties for auditors to find the actual evidence files and prolonged the audit schedule by necessitating requests for clarifications and re-submittal of additional evidence. Current status As per the December 2011 CWG meeting, we now provide a index template filtered by entity type as an attachment to the notification letter. A generic (non-filtered) copy is provided upon request. Possible Constraints The auditors need the entity to point to the evidence document/page for each applicable requirement. 8 Bad index of evidence files Our thoughts Since the need for an index depends on our decision to use RSAWs or not, we propose that 6, 8 and 9 be discussed as one issue. Group discussionвЂ¦ Does this work? 8 Bad index of evidence files SUMMARY The CWG: вЂў Agreed to discuss this issue with issue #9 Audit guideline - 7.6: Submit audit evidence 9 Audit index VS use of RSAWs Background Issue numbers 6 and 8 outlined the difficulties that the auditors were having in finding evidence documents because of bad indexes. Entities suggested that perhaps the RSAW could be submitted. A pilot project enabling submission of both the RSAW and index (as back-up) was provided to requesting entities for 2012 Q1 and Q2. Current status As per the December 2011 CWG meeting, we now provide an index template filtered by entity type as an attachment to the notification letter. A generic (non-filtered) copy is provided upon request. As a pilot project, RSAW submission with index back-up is permitted when requested. Possible Constraints The intent is that the index or RSAW be used to point the auditor to the evidence document/page related to the requirement/measure of the standard. For efficiency, only one method should be used. 9 Audit Index VS use of RSAWs Our thoughts The pilot project of using RSAWs was quite successful. вЂў There were no major difficulties in finding evidence. вЂў Most submitted RSAWs used imbedded document links which proved to be quite efficient. вЂў In one case the link did not work however the document was found using the index back-up. Alternatively an additional request could have been made. вЂў The вЂњDescriptionвЂќ column was sometime used to give or state evidence. The intent of the column is to describe the evidence (such as paragraph or page number). вЂў The purpose of using the RSAW should be the same as the index вЂ“ to point the auditor directly to the evidence. The advantage of the RSAW is that the requirement, measure and assessment step are all aligned in one document and page row. Group discussionвЂ¦ Index VS RSAW? Impact and broadness? 9 Audit Index VS use of RSAWs SUMMARY The CWG: вЂў Agrees with using the RSAW instead of the index template вЂў Notes: вЂў Language in RSAW could be clearer вЂў Separate the request from auditor functions o Side note: New RSAW are formatted this way Audit guideline - 7.7: Review of completeness 10 (Review) Initial review too detailed Background (Internal issue) The completeness review went beyond a simple check that all documents were received and usable. Some assessment was being performed during the completeness review and then repeated in more detail during the assessment phase. Some entities were confused by the similar nature of requests during the completeness review and assessment phases. Current status As discussed at the December 2011 CWG: вЂў The guideline will be changed to state вЂњAdministrative reviewвЂќ so as to align with language used in self-certification вЂў We will limit the вЂњadministrative reviewвЂќ to the intended checks вЂў We will better communicate the intent through training workshops Audit guideline - 7.7: Review of completeness 11 How can we help entities better manage information requests? Background Some participants have asked us to give them an idea of when to expect additional information requests in order to better plan SME resources during the audit assessment phase. Current status Although auditors try to provide an estimate of when additional requests will be available, it is not part of the process to provide an audit assessment schedule Possible Constraints Information request timelines may depend on volume of material which is not known until the initial вЂњadministrative reviewвЂќ is completed. Once the volume is known, a tentative schedule of when additional requests for each standard type is possible вЂ“ however, there are too many variables to be able to determine in advance when the draft or final report will be completed. 11 Need to establish an additional request schedule Our thoughts вЂў Establishing a schedule of when information requests will help entities and us to better prepare internal resources вЂў Any schedule would be on a tentative basis and could not be guaranteed Possible inclusions in process: вЂў After the вЂњadministrative reviewвЂќ, the AESO will provide a tentative schedule of when the entity can expect information requests. вЂў The entity would provide any possible constraints вЂў The AESO will make вЂњbest effortвЂќ to accommodate entity needs. Group discussionвЂ¦ Work for you? Impact and broadness? 11 Need to establish an additional request schedule SUMMARY The CWG: вЂў Agrees with the approach of providing a schedule for additional requests: вЂў Schedule is a best guess вЂў Group by standard area вЂў Possibly update tentative schedule on known changes Ex: Where it conflicts with the constraints Audit guideline - 7.7: Review of completeness 12 Additional request content is sometimes unclear Background A participant stated that additional requests are not always clear and require additional follow-up to be understood. This leads to difficulty in tracking requests, confusion and inefficiencies. Further to this it is not clear how an entity can request clarifications. Current status Process is not sufficiently detailed. Possible Constraints Requests are made so that the auditor is satisfied that all available evidence has been received. Multiple requests may still be necessary. 12 Additional request process not efficient Our thoughts Establishing a more detailed process may help clarify request content. вЂў Auditors will use a template for requests вЂў Template to include all key information вЂў Standard / Requirement вЂў Background of request and any related documents вЂў Entities to use the template to respond вЂў Space provided for response вЂў Space provided to attach documents Group discussionвЂ¦ Work for you? Impact and broadness? 12 Additional request process not efficient SUMMARY The CWG: вЂў Agrees that using a template is good вЂў Entity should communicate any need for clarifications and the AESO will re-word the request if required. Audit guideline - 7.11: Review draft audit report 13 Extent of commenting on re-issued draft report Background The guideline allows for re-issue of the draft report: вЂњIn the event a re-evaluation of previously submitted audited evidence is deemed appropriate, and results in revised audit findings or significant changes to the draft audit report, a new draft audit report will be issued and the registered market participant will be provided ten business days to review and submit their comments in writing to the compliance monitor.вЂќ It is unclear however as to what extent comments may be provided. Current status No changes have been made to section 7.11 of the guideline. Possible Constraints None identified 13 Extent of commenting on Re-issued draft report Our thoughts Comments on a re-issued draft report should be limited to items that have changed. The guideline wording would be changed to: вЂњIn the event a re-evaluation of previously submitted audited evidence is deemed appropriate, and results in revised audit findings or significant changes to the draft audit report, a new draft audit report will be issued and the registered market participant will be provided ten business days to review and submit their comments on the changes in writing to the compliance monitor.вЂќ Group discussionвЂ¦ Work for you? Impact and broadness? 13 Extent of commenting on Re-issued draft report SUMMARY The CWG: вЂў Agrees with the approach of adding вЂњon the changesвЂќ to this section of the audit guideline. Next steps вЂў We will send power point summaries to all CWG members вЂў Create solutions based on todayвЂ™s results вЂў Set next CWG meeting вЂў End July / Early August вЂў Date: Will send out survey to members on possible dates вЂў Time: Afternoon, mid-week is good. вЂў Purpose вЂў AESO to present solutions based on first meeting input вЂў CWG to refine Thank You Any questions or comments?