Typical Challenges

Typical challenges

Here are two challenges that often come up when you try to take an AI prototype to production. Use them to get a feel for what we look at, and what we decide, in an assessment.

What follows are examples of likely situations, not real client cases. We show no results or figures. Once we have real engagements, we will publish them with the client's permission.

Scenario 1

An internal generative-AI tool (RAG) stalls on answer accuracy and access permissions

A common situation

A team has prototyped an internal Q&A chatbot loaded with the company's technical documents, but some answers are wrong. Access can't be controlled by role or department, so the tool can't be rolled out company-wide.

What we'd look at in an assessment

  • How document splitting and retrieval design affect answer accuracy
  • How answer correctness is evaluated and checked
  • Whether access permissions are enforced only inside the app
  • Logging, monitoring and handling of confidential information

An example of how to proceed

  • Consider a design that enforces permissions at the point data is retrieved
  • Compare ways to connect to the existing authentication platform (such as a corporate directory)
  • Build a set of evaluation questions so improvements can be compared before and after

Scenario 2

You want to add AI features to an existing system, but its structure gets in the way

A common situation

A company wants to add AI-powered analytics to its main production product. But the existing structure is complex, the impact of changes is hard to predict, and development drags on while bugs increase.

What we'd look at in an assessment

  • Dependencies between the existing code and the data structures
  • The impact of changes, and what the tests cover
  • Embedding the AI feature directly versus separating it
  • The flow of data, and what can be referenced safely

An example of how to proceed

  • Consider connecting the AI feature through an API without touching the existing system
  • Start with read-only integration and expand while checking the impact on core operations
  • At each stage, share acceptance criteria and remaining issues

Start with your own case

Similar situations can have different causes. Tell us about yours, and we'll propose next steps case by case.

Contact us