Choosing a security awareness vendor is a procurement decision about operating a human-risk program, not a hunt for the longest course catalogue. The right platform should help the organisation enrol the right people, practise realistic decisions, respond to reports, support managers and show evidence without creating a second manual process for the security team.
This guide gives Australian organisations and MSPs a practical vendor evaluation method for 2026. It focuses on fit, evidence, operating effort and contract risk rather than unverified promises about a universal reduction in incidents.
TL;DR
- Define the behaviours, audiences and evidence required before asking vendors for a demonstration.
- Score the end-to-end workflow: enrolment, learning, phishing practice, reporting, remediation and review.
- Test data handling, integrations, access controls, support and exit terms as seriously as content quality.
- Run a time-boxed pilot with representative users and the same measures every vendor must answer.
- Choose the platform that makes the required program repeatable, not the one with the most features on a slide.
Start with the outcome
Write the procurement outcome in one sentence: the organisation needs a repeatable way to help defined groups recognise risky requests, take a safe action, report concerns and give leaders evidence of progress. Then name the workflows where that matters. Examples include supplier payment changes, unexpected MFA prompts, sensitive file sharing, customer-account requests and privileged access.
A cyber security gap assessment can help structure the starting point. Use its findings, internal incidents, helpdesk questions, audit observations and business changes to identify the behaviours the vendor must support. Do not let the vendor define the problem by showing its product first.
The Essential Eight is an Australian baseline of eight mitigation strategies. A training platform can support the human actions around controls such as MFA, application control and privileged access, but it cannot implement the controls. Put the technical and process dependencies beside the learning requirements so the tender does not promise what training cannot deliver.
Step 1: define the audiences and journeys
List every audience that will use the program:
- Employees who need the baseline learning and reporting route.
- New starters who need training during onboarding.
- Contractors and temporary workers who may have different access or due dates.
- Finance, HR, customer teams and other groups with role-specific decisions.
- Privileged users who need a higher level of account and access awareness.
- Executives and board members who approve payments, disclosures or urgent actions.
- Managers who must act on overdue work and support targeted remediation.
For each audience, write the journey from start to finish. How is the person added? Which lesson is assigned? What happens when they complete, fail or miss the due date? How do they report a suspicious message? What can their manager see? What happens when they leave or change role? A vendor that answers only the first assignment step is not demonstrating a program.
The Cyber Aware training page describes the platform’s training approach. Treat any vendor description as a claim to test against the organisation’s own journeys, not as evidence that the workflow will fit without configuration.
Step 2: build a weighted scorecard
Use a scorecard before demonstrations begin. A practical starting point is:
- Program fit: 25 points. Audience paths, role-based content, onboarding, reminders and remediation.
- Practice and reporting: 20 points. Phishing scenarios, reporting route, response workflow and comparable measures.
- Administration: 15 points. User lifecycle, integrations, delegation, support for multiple clients and export.
- Security and privacy: 15 points. Data locations, access controls, retention, subprocessors and incident obligations.
- Evidence and measurement: 10 points. Definitions, dashboards, history, audit evidence and useful exports.
- Commercial and exit terms: 10 points. Seat rules, renewal, price changes, implementation and data portability.
- Accessibility and experience: 5 points. Usability, language, accessibility and mobile or remote access needs.
Change the weights when the organisation’s risk requires it, but do not remove the categories. Give every requirement a pass condition. For example, support for leavers means a documented process that removes access within the organisation’s required window, not a checkbox labelled user management.
Step 3: test the content against real work
Ask each shortlisted vendor to show three examples using the organisation’s scenarios. One should be a baseline message, one should be a role-specific request and one should involve reporting or escalation. The demonstration should show the learner view and the administrator view.
Look for plain language, realistic Australian context, clear actions and an explanation of what happens after reporting. Avoid content that relies on fear, embarrassment or gotcha questions. A course can be technically accurate and still fail if people cannot remember the next action under pressure.
Ask how content is reviewed, dated and retired. Ask how the organisation can add policy-specific guidance without creating contradictory instructions. Confirm whether completion, quiz results and assignments can be reported separately; combining them into one green status can hide a knowledge or process problem.
Step 4: evaluate phishing practice safely
Phishing practice is valuable when it rehearses a reporting route and produces evidence that can be compared over time. The Cyber Aware phishing page describes simulated campaigns and follow-up learning. In a procurement test, ask the vendor to show how a campaign is created, approved, targeted, scheduled, stopped and reviewed.
Ask these questions:
- Can campaigns be limited to a defined cohort and excluded for a documented reason?
- Can the organisation use different scenarios for finance, HR, executives and privileged users?
- What does the learner see after clicking or reporting?
- Can the program distinguish clicks, credential submissions, reports and repeat events?
- Are managers shown useful exceptions without exposing unnecessary personal detail?
- Can the organisation export a stable result with the campaign date, audience and definitions?
- What controls prevent a simulation from resembling a real high-impact transaction?
Do not select a vendor because it offers the most aggressive simulation. Select one that can practise the right decision without damaging trust or overwhelming the response team.
Step 5: inspect reporting and measurement
A vendor should explain what each measure means and what decision it supports. Start with a small set: completion by due date, overdue rate, click rate, report rate, time to report and relevant helpdesk or incident themes. Ask whether the platform preserves the cohort, date range and denominator so that month-to-month comparisons remain honest.
The human risk reporting approach is useful as a benchmark for the type of view a security or MSP team may need: learning activity, quiz outcomes and phishing behaviour brought together for action. A report is only useful if a manager can understand the exception, an owner can respond and the organisation can see whether the action changed the next result.
Ask for a sample board or client report using fictional data, then ask the vendor to explain every number. Reject dashboards that look impressive but cannot state the audience, period, calculation or action behind a measure.
Step 6: verify administration and integrations
Administration cost is often the difference between a program that runs every month and one that becomes an annual scramble. Test the user lifecycle with a small set of scenarios: a new starter, a leaver, a role change, a contractor, a duplicate account and a person on extended leave. Ask who performs each action and how failures are detected.
Confirm the integrations the organisation actually needs. Do not award points for a long list of logos. Ask whether the connection supports enrolment, role or group mapping, leaver removal, assignment and reporting, and whether the process can be tested without sending the whole directory.
If an MSP will operate the platform, test client separation, delegated administration, white-labelling, client-level reporting and the ability to prevent one client’s data appearing in another client’s view. Ask how a client leaves the MSP relationship and receives its data.
Step 7: review security, privacy and contract evidence
The ACSC Guidelines for procurement and outsourcing, first published and updated in June 2026, says organisations should consider cyber supply-chain risks during procurement and identify jurisdictional, governance, privacy and security risks associated with suppliers. It also recommends a shared responsibility model that clearly states the responsibilities of supplier and customer.
Ask each vendor for evidence and record the answer:
- Where are learner records, campaign results and reports stored and processed?
- Which subcontractors can access the data, and how are changes communicated?
- What security obligations, incident notifications and response times appear in the contract?
- How are administrator access, authentication, logging and privileged support controlled?
- What is the retention period, and can the organisation delete or export its data?
- What happens to records at termination, including backups and client copies?
- Can the organisation verify important security commitments during the relationship?
NIST SP 800-161 Rev. 1 Update 1, Cybersecurity Supply Chain Risk Management Practices for Systems and Organizations, describes identifying, assessing and mitigating cybersecurity supply-chain risk for products and services. Use it to frame the diligence questions; it does not provide a pass or fail certificate for a training platform.
Step 8: run a fair pilot
A pilot should answer a decision, not generate free consulting. Use a representative sample that includes a manager, a new starter, a high-risk role and an administrator. Give every shortlisted vendor the same audience, timeframe, scenarios and success measures.
A useful pilot has four phases:
- Setup: configure the audience, reporting route, baseline lesson and one role-specific assignment.
- Use: let participants complete the lesson and report a controlled test message.
- Review: compare completion, quiz results, clicks, reports, time to report and administrator effort.
- Decision: record what passed, what required manual work and what remains unproven.
Do not use a tiny pilot to claim organisation-wide impact. Use it to expose workflow friction, unclear instructions, missing integrations and data-handling questions before a contract is signed.
Step 9: compare total operating cost
Ask for the full first-year and recurring-year cost. Include seats, setup, content, integrations, support, white-labelling, phishing practice, reporting, implementation time and renewal terms. Then estimate internal administration using the tasks observed during the pilot.
The Cyber Aware comparison page can be one input to a shortlist, but procurement should compare the same requirements across every option. A cheaper licence may cost more if the security team spends hours reconciling users or rebuilding evidence. A premium feature may be waste if it is not tied to a defined behaviour or audience.
Check commercial details that are easy to miss: minimum seats, treatment of inactive users, automatic renewal, annual price increases, currency, taxes, support hours, implementation boundaries and exit assistance. Get material answers into the contract rather than leaving them in a sales call.
Red flags that should stop the evaluation
- The vendor cannot show the learner and administrator experience in the same demonstration.
- The report uses risk scores without explaining the inputs, denominator or time period.
- The vendor treats course completion as proof that the organisation is secure.
- Data location, subprocessors, retention or exit terms are unclear.
- The vendor will not run a representative pilot or answer the same questions in writing.
- Simulations are designed to shame people rather than improve reporting and response.
- Client separation or delegated administration is asserted but not demonstrated.
A 30-day procurement plan
In days 1–5, confirm the risk statement, audiences, required workflows and scorecard. In days 6–12, issue the same scenario-based questions to the shortlist. In days 13–20, run demonstrations and review security, privacy and contract evidence. In days 21–26, run the pilot and record administration effort. In days 27–30, score the evidence, document gaps and make the recommendation.
NIST SP 800-50 Rev. 1, Building a Cybersecurity and Privacy Learning Program, supports a lifecycle approach that can be adapted as organisational needs change. Treat the procurement decision as the start of that lifecycle: assign an owner, set a review cadence and define what will cause the program to change.
FAQ
What is the most important question to ask a security awareness vendor?
Ask the vendor to demonstrate the full journey for one real behaviour: enrolment, lesson, practice, report, remediation, manager action and evidence. The answer reveals more than a feature list.
Should content or platform capability carry more weight?
Neither should win automatically. Content must be relevant and understandable, while the platform must make delivery, practice, reporting and follow-up repeatable. Score both against the organisation’s defined outcomes.
How many vendors should take part in a pilot?
Pilot only the shortlist that has passed the written requirements and diligence review. A smaller, fair comparison with consistent scenarios is more useful than a large demonstration parade.
What security evidence should a vendor provide?
Request evidence relevant to the data and service: access control, authentication, logging, data location, subprocessors, incident handling, retention, deletion, export and contractual responsibilities. The exact evidence depends on the organisation’s risk and procurement policy.
Is an awareness platform suitable for an MSP?
It can be, if it supports client separation, delegated administration, white-labelling, client-level reporting, user lifecycle and a clear client exit process. Test those workflows instead of assuming a multi-tenant claim is sufficient.
How should procurement judge phishing simulations?
Judge whether they practise the intended reporting decision, protect trust, provide comparable measures and trigger useful follow-up. More difficult or more frequent simulations are not automatically better.
One last thing
The best vendor is the one the organisation can operate consistently after the sales team leaves. Make the decision on demonstrated workflow, verified evidence, total operating effort and clear responsibility—not on a feature count that nobody will use.
Sources
- ASD Guidelines for procurement and outsourcing, first published and updated June 2026.
- NIST SP 800-161 Rev. 1 Update 1: Cybersecurity Supply Chain Risk Management Practices for Systems and Organizations, published May 2022 with updates as of November 2024.
- NIST SP 800-50 Rev. 1: Building a Cybersecurity and Privacy Learning Program, published September 2024.
- ASD Essential Eight, Australian baseline guidance.