Case studies

Supporting AI-readiness in Australia

Who: Australian Government’s Digital Transformation Agency (DTA)

What they did: Develop policies and tools to support effective and safe use of AI technology, including:

• Policy for the responsible use of AI in government
• Technical standard for government’s use of artificial intelligence
• AI model clauses

What worked well: The DTA meets staff where they are by directly embedding their guidance on good procurement practices for AI technology in go-to resources like their Digital Marketplace, 2 panel agreement, and the BuyICT request for quote procurement workflow. Although the DTA also promotes its materials in other ways, such as through blog posts, they’ve seen that the tactic of integrating the guidance in something that agencies use on a regular basis to be a successful way to support adoption.

DTA also keeps their policy documents live, so that they are able to update them on a regular basis and stay current with the latest policy and technology developments.

What was harder: At the time, use of the templates was mandatory, and felt overly ridged to SUSESO, especially the balance towards price and competition over criteria around vendor capacity and experience around important issues like data protection and model bias, transparency and explainability. These templates have been updated based on SUSESO’s and others’ experience, and new legislation, and are now optional.

Lesson learned: AI procurement templates cannot solve for organizational AI-readiness. For example, Chile’s templates do not address how AI should be used in collaboration with human decision-making, and what acceptable risk looks like when it comes to leveraging AI for making decisions that matter for people’s lives.

Outcome-driven contracting success in the United States

Who: Centers for Medicare and Medicaid Services (CMS)

What they bought: What they bought: AI/ML services to streamline operations, as well as some maintenance of existing legacy tools, from Skyward IT Solutions, LLC; the resulting tools include a chatbot for the CMS' Office of Information Technology team that has reduced the time it takes to draft tickets with a user story from 15-20 minutes down to one minute, as well as a procurement tool for the Office of Acquisition and Grants Management that enables faster labor category analysis, which is now saving the CMS $3 million a month.

Procurement procedure: Challenge-based procurement through an open and competitive process

What worked well: The high-level, open-ended, outcome-driven contract focused on problems rather than dictating solutions. Given the rapid pace of AI development, there are now AI capabilities that were not available when the RFP was drafted or the contract was signed. The flexible contract has allowed the vendor to deliver the most cutting-edge solutions possible to CMS.

What was harder: Institutions are still learning about AI, how it works, and what is required to get the most from this technology, and sometimes building this awareness falls on the vendor. For example, there is still a lack of understanding of the value of a specialized versus generic AI solutions. Generic models can be useful, but are not a one-size fits all solution. Narrow government use cases can require highly specific knowledge management, and be a better fit for specialized solutions.

Lesson learned: For a contracting approach like this to be successful, the government must be willing to trust their vendor partner.

Ethical and effective procurement of AI for medical claims in Chile

Who: Chile’s Ministry of Labor - Department of Social Security Superintendence (SUSESO)

What they bought: Pilot technology to speed up claims resolutions, including a medical claims machine learning model and mental health claims machine learning model audit.

Procurement procedure: Open and competitive processes

What worked well: ChileCompra, the national procurement agency, worked with the University of Adolfo Ibáñez’s School of Government’s Ethical Algorithms Project to develop standard templates for vendor bids on data science and AI projects.

SUSESO used these then-mandatory templates for its procurement processes. SUSESO leadership feels that the templates helped push vendors to provide impact assessments and documentation around responsible AI requirements, as well as build internal awareness around what criteria to consider when buying AI, especially around bias, transparency, and data protection.

What was harder: Institutions are still learning about AI, how it works, and what is required to get the most from this technology, and sometimes building this awareness falls on the vendor. For example, there is still a lack of understanding of the value of a specialized versus generic AI solutions. Generic models can be useful, but are not a one-size fits all solution. Narrow government use cases can require highly specific knowledge management, and be a better fit for specialized solutions.

Lesson learned: For a contracting approach like this to be successful, the government must be willing to trust their vendor partner.

Buying through a framework agreement in Italy

Who: The central procurement agency of Italy, CONSIP, on behalf of the entire Italian government.

What they bought: AI application services and support for the Italian National Health System (SSN), valued at $770 million over the next 6 years.

Procurement procedure: RFP framework agreement for AI application services.

What worked well: Using a simplified procurement model to enable AI application services to be directly ordered from vendors, while Gen AI LMM models and cloud infrastructure/services are made through separate call-offs. This leverages existing government frameworks for cloud services, eliminating the need to specify cloud infrastructure and AI service requirements.

What was harder: Identifying the right consortium of companies (Temporary Association of Companies) to cover the full spectrum of services, including areas like AI, ML, Natural Language Processing/Understanding, Big Data, and SaaS solutions on metadating and ontology creation for the health sector.

Lesson learned: Despite the inherent complexity of AI procurement, the technical proposal under evaluation was limited to 40 pages to cover all critical aspects of AI procurement. This included regulatory compliance, security/privacy, bias/discrimination mitigation, quality and reliability in terms of transparency and explainability, IP rights, skills and know-how, continuous performance monitoring, integration/interoperability, and cost-effectiveness.